AV testing practices questioned

Posted by   Virus Bulletin on   Aug 14, 2007

Professional and amateur tests criticised.

Last week, IT industry commentator and renowned anti-anti-virus writer Robin Bloor released a typically inflammatory article implying widespread corruption in anti-virus testing, suggesting that testing organisations within the industry are complicit in rigging test results to show AV products in a good light. This week, results of a supposedly independent test run live at a conference have been widely criticised for inaccuracies and sloppy methodology.

Bloor's article, entitled 'Is AV product testing corrupt?', quotes an anonymous contact 'high up in one of the IT security companies' as being 'suspicious' of results released by highly respected testing centre AV-test.org, and goes on to quote at length from another anonymous source, discussing the problem of test collections containing samples of dubious provenance and uncertain relevance. The article suggests that AV companies, providing samples to testing bodies for inclusion in test sets, routinely bias results by sending items only detected by their own products, and that test sets are riddled with corrupt, non-functional or simply non-malware samples.

At the LinuxWorld Expo in San Francisco last week, open-source gateway software producer Untangle presented an 'anti-virus Fight Club', testing a selection of products live on stage against a set of 35 samples gathered from the expo audience as well as the organisation's own inboxes. Kaspersky, Norton and open-source ClamAV were found to detect 100% of the samples, while others did less well, including one which scored less than 6% overall.

Since the release of the results, several commentators have pointed out flaws in the running of the test, not least the small sample set, the comparison of wildly different product types and errors in the settings used. The company running the test, as a vendor of a product using ClamAV, also appears to have an interest in the results, and having made its test set freely available online, risks charges of distributing malware.

'Testing AV products is an enormously complex and difficult business,' said John Hawes, Technical Consultant at Virus Bulletin. 'Amateur tests always run the risk of producing erroneous results, due to lack of experience in designing proper test procedures, in operating a wide range of products and, most importantly, in creating and maintaining a malware collection. The serious testing organisations spend enormous amounts of time and energy ensuring sets contain only valid samples, a process which was the central focus of the recent testing symposium in Iceland. The likelihood that the massive collections used by bodies like AV-Test or AV-Comparatives could be biased by individual vendors providing samples favourable to their own products seems pretty remote.'

'Of course, these days there is a lot more to security software than the ability to detect known malware in on-demand scans,' continued Hawes. 'The problems presented by testing heuristic and behavioural detection, as well as properly comparing protection offered by multi-level suites, are currently being worked on with great urgency across the industry, and hopefully firm and comprehensive methodologies for a wide range of testing will be agreed on soon. The question remains as to whether the average user can be persuaded to understand and take an interest in more complex and detailed results, as opposed to the simple percentage scores which the popular press currently demands.'

Bloor's story is here and is also carried in The Register here. Details of Untangle's LinuxWorld test start here and continue over several pages. A McAfee blog entry detailing problems spotted with the Untangle test, and linking to several in-depth articles on the difficulties of testing, is here, with some less sober commentary from Randy Abrams of ESET here and further analysis from independent security researcher David Harley available (in PDF format) here.

The latest comparative review carried out by a magazine's own in-house lab, testing the suitability of products for use in governmental institutions and focussing mainly on usability, price and scanning speed rather than detection rates, is in Government Computer News here.

Posted on 14 August 2007 by Virus Bulletin

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest posts:

VB2018 paper: From Hacking Team to hacked team to…?

Today we publish the VB2018 paper and video by ESET researcher Filip Kafka, who looked at the new malware by Hacking Team, after the company had recovered from the 2015 breach.

The spam that is hardest to block is often the most damaging

We see a lot of spam in the VBSpam test lab, and we also see how well such emails are being blocked by email security products. Worryingly, it is often the emails with a malicious attachment or a phishing link that are most likely to be missed.

Throwback Thursday: We're all doomed

Mydoom turns 15 this month, and is still being seen in email attachments. This Throwback Thursday we look back to March 2004, when Gabor Szappanos tracked the rise of W32/Mydoom.

VB2019 call for papers - now open!

Have you analysed a new online threat? Do you know a new way to defend against such threats? Are you tasked with securing systems and fending off attacks? The call for papers for VB2019 is now open and we want to hear from you!

VB2018 paper: Unpacking the packed unpacker: reversing an Android anti-analysis library

Today, we publish a VB2018 paper by Google researcher Maddie Stone in which she looks at one of the most interesting anti-analysis native libraries in the Android ecosystem. We also release the recording of Maddie's presentation.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.