AV testing practices questioned

Posted by   Virus Bulletin on   Aug 14, 2007

Professional and amateur tests criticised.

Last week, IT industry commentator and renowned anti-anti-virus writer Robin Bloor released a typically inflammatory article implying widespread corruption in anti-virus testing, suggesting that testing organisations within the industry are complicit in rigging test results to show AV products in a good light. This week, results of a supposedly independent test run live at a conference have been widely criticised for inaccuracies and sloppy methodology.

Bloor's article, entitled 'Is AV product testing corrupt?', quotes an anonymous contact 'high up in one of the IT security companies' as being 'suspicious' of results released by highly respected testing centre AV-test.org, and goes on to quote at length from another anonymous source, discussing the problem of test collections containing samples of dubious provenance and uncertain relevance. The article suggests that AV companies, providing samples to testing bodies for inclusion in test sets, routinely bias results by sending items only detected by their own products, and that test sets are riddled with corrupt, non-functional or simply non-malware samples.

At the LinuxWorld Expo in San Francisco last week, open-source gateway software producer Untangle presented an 'anti-virus Fight Club', testing a selection of products live on stage against a set of 35 samples gathered from the expo audience as well as the organisation's own inboxes. Kaspersky, Norton and open-source ClamAV were found to detect 100% of the samples, while others did less well, including one which scored less than 6% overall.

Since the release of the results, several commentators have pointed out flaws in the running of the test, not least the small sample set, the comparison of wildly different product types and errors in the settings used. The company running the test, as a vendor of a product using ClamAV, also appears to have an interest in the results, and having made its test set freely available online, risks charges of distributing malware.

'Testing AV products is an enormously complex and difficult business,' said John Hawes, Technical Consultant at Virus Bulletin. 'Amateur tests always run the risk of producing erroneous results, due to lack of experience in designing proper test procedures, in operating a wide range of products and, most importantly, in creating and maintaining a malware collection. The serious testing organisations spend enormous amounts of time and energy ensuring sets contain only valid samples, a process which was the central focus of the recent testing symposium in Iceland. The likelihood that the massive collections used by bodies like AV-Test or AV-Comparatives could be biased by individual vendors providing samples favourable to their own products seems pretty remote.'

'Of course, these days there is a lot more to security software than the ability to detect known malware in on-demand scans,' continued Hawes. 'The problems presented by testing heuristic and behavioural detection, as well as properly comparing protection offered by multi-level suites, are currently being worked on with great urgency across the industry, and hopefully firm and comprehensive methodologies for a wide range of testing will be agreed on soon. The question remains as to whether the average user can be persuaded to understand and take an interest in more complex and detailed results, as opposed to the simple percentage scores which the popular press currently demands.'

Bloor's story is here and is also carried in The Register here. Details of Untangle's LinuxWorld test start here and continue over several pages. A McAfee blog entry detailing problems spotted with the Untangle test, and linking to several in-depth articles on the difficulties of testing, is here, with some less sober commentary from Randy Abrams of ESET here and further analysis from independent security researcher David Harley available (in PDF format) here.

The latest comparative review carried out by a magazine's own in-house lab, testing the suitability of products for use in governmental institutions and focussing mainly on usability, price and scanning speed rather than detection rates, is in Government Computer News here.

Posted on 14 August 2007 by Virus Bulletin



Latest posts:

AfricaHackon 2019: a great event and a reminder that security is global

Last week, VB Editor Martijn Grooten travelled to the Kenyan capital Nairobi to speak at the 6th edition of the AfricaHackon event.

Virus Bulletin researcher discovers new Lord exploit kit

Still in-development kit thus far only targets Flash Player vulnerabilities

VB2019 call for last-minute papers opened

The call for last-minute papers for VB2019 is now open. Submit before 1 September to have your abstract considered for one of the nine slots reserved for 'hot' research.

Nominations opened for sixth Péter Szőr Award

Virus Bulletin is seeking nominations for the sixth annual Péter Szőr Award.

Haroon Meer and Adrian Sanabria to deliver VB2019 closing keynote

New additions to the VB2019 conference programme include a closing keynote address from Thinkst duo Haroon Meer and Adrian Sanabria and a talk on attacks against payment systems.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.