Respecting the testing

2006-09-01

John Hawes

Virus Bulletin, UK
Editor: Helen Martin

Abstract

'Competition for good test results, and so for respect, trust and strong sales, feeds development and innovation.' John Hawes, Virus Bulletin, UK


Last month, ConsumerReports.org (CR), the online arm of the US Consumers' Union society, announced proudly to the world its decision to create 5,500 virus 'variants', as part of an extensive test of anti-virus products. No details were provided as to what these were variants of, how they were created or verified, or how they were put to use in the tests. All that is known is that the malware was provided by ISE (Independent Security Evaluators), whose president Avi Rubin disassociated himself from the tests.

The initial reaction within the AV industry was a slow, sad shaking of heads, and raised eyebrows of disbelief. Sensible voices pointed out the flawed methodology of the tests and the availability of similar test results from specialized and respected test centres running retrospective testing. These organizations focus on the same issues as those raised by CR, but they use existing, real-world viruses. Pessimistic members of the AV community feared the escape of the new variants into the wild, and pondered the legal implications for their creators should they cause any damage. Analysts complained at the volume of extra work that looms, once the fat chunk of new malware finally reaches their desks for inspection and identity creation (one of the few statements issued by CR has suggested that handing the malware over to industry experts would be 'a good idea'). Minds were cast back to similar scandals of the past, to the tests carried out by CNET using the Rosenthal 'simulated' viruses, and to the infamous University of Calgary 'virus-writing' course.

Since then, the issue has mushroomed into a fizzing cloud of counter-accusations. Backlash against the virus creators' critics has mainly taken the form of accusations against the AV industry of hiding the reliance of AV products on signature-based detection and of hyping the efficacy of heuristic methods (of course, the old 'they-write-all-the-viruses-themselves' chestnut has cropped up here and there too). 'They're telling you they have all this heuristic capability, but the best they can do is 50 per cent. That's nothing; that's terrible.' So cried Peter Firstbrook, head of the cyber security division of marketing palmist Gartner, fresh from compiling the highly lucrative 'magic quadrant' report on the AV industry. User faith in virus protection is being battered by a hail of abuse.

Testing is important. Competition for good test results, and so for respect, trust and strong sales, feeds development and innovation. Flashy logos and catchy slogans may capture the eye and the ear, but without the credibility given by proven effectiveness, no product can hope to thrive. Dissemination of test results also helps users, allowing them to judge the performance of their product, and to demand better where it is available. To achieve these goals, testing must be credible, it must be transparent, verifiable and accountable.

VB, along with several other specialized and recognized testing organizations, provides a vital service, both to those within the industry and to their customers, ensuring that security software performs as well as it can. To provide these services the testing organizations rely on the community they serve, and abide by its ideologies and beliefs. One of the most strongly held convictions throughout the industry is that creating viruses is never justified. Those who do so are forever beyond the pale, barred from employment, as they are from respect and trust, by their fellows. In conniving in the creation of viruses, CR has damaged its credibility as surely as if it had been involved in any other criminal activity.

Here at VB we hope, in the near future, to improve and expand our own testing, to include an ever-growing variety of threats; with spyware on the horizon, the shadowy territory of rootkits and the legal minefield of adware lie ahead. We would like, at some point, to be able to include an element of retrospective testing into our service. In order to do all this successfully and properly, we rely on the trust and respect of our readers and of the industry we study and report on, and we will certainly not be hiring any virus writers.

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest articles:

VB2018 paper: From Hacking Team to hacked team to...?

In this paper (presented at VB2018), Filip Kafka looks at the resurfaced Hacking Team spyware, and at what has changed since the company behind it faced a number of prominent hacks.

Throwback Thursday: We're all doomed

When a daily sports paper compares a national soccer crisis with the spread of an Internet worm, you know that the worm has had an enormous impact on everyday life. In March 2004, Gabor Szappanos tracked the rise of W32/Mydoom.

VB2018 paper: Unpacking the packed unpacker: reversing an Android anti-analysis native library

This paper analyses one of the most interesting anti-analysis native libraries we’ve seen in the Android ecosystem. No previous references to this library have been found.The anti-analysis library is named ‘WeddingCake’ because it has lots of layers.

VB2018 paper: Draw me like one of your French APTs – expanding our descriptive palette for cyber threat actors

When it comes to the descriptive study of digital adversaries, we’ve proven far less than poets. Currently, our understanding is stated in binary terms: ‘is the actor sophisticated or not?’. Juan Andres Guerrero-Saade puts forward his views on how we…

VB2018 paper: Office bugs on the rise

It has never been easier to attack Office vulnerabilities than it is nowadays. In this paper Gabor Szappanos looks more deeply into the dramatic changes that have happened in the past 12 months in the Office exploit scene.


Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.