Who will test the testers?

David Harley and Andrew Lee ESET

The anti-malware industry has been plagued since its earliest days by one poorly designed comparative test after another. In 2007, some of the best anti-malware researchers, comparative testers, and product certification specialists took the first steps towards raising product testing standards with the formation of a group specifically focused on establishing standards and methodologies, educating both consumers and testers in discriminating between good and bad practice, and providing objective analyses of current testing practices. This paper summarizes current initiatives by the Anti-Malware Testing Standards Organization (AMTSO) and other groups, but also considers next steps, going beyond objectifying methodology, educational issues and blowing away the fog of misinformation and fallacy, to the next level.

Underlying these vital issues is a question: is it possible to make testers and certifying authorities more accountable for the quality of their testing methods and the accuracy of the conclusions they draw, based on that testing?

This paper attempts to answer that question.


We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.