AV industry commments on anti-malware testing


Randy Abrams

Eset, USA
Editor: Helen Martin


'Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users' Randy Abrams, Eset.

In the fine tradition of the pioneers of the anti-virus industry, the 1st International Antivirus Testing Workshop was conceived and held in Reykjavik, Iceland last month. Michael St. Neitzel, formerly of Microsoft and Eset, now working for Frisk, had seen one too many unacceptably bad AV tests and decided it was time to bring AV researchers and testers together to try to improve the state of malware testing.

Researchers have beaten up on testers for years with little discernable result, so the notion of such a meeting to improve the status quo may seem a little quixotic unless one realizes that the opponent the AV industry faces is not a windmill, but rather, in the words of Dr Klaus Brunnstein, its Siamese twin.

The presentations were interesting and can be found at http://www.f-prot.com/workshop2007/, however the majority of value came in the discussions that followed the presentations.

A presentation modestly entitled 'Building & leveraging white database for antivirus testing' by Mario Vuksan from Bit9 was the sleeper. The presentation exposed not only the complexities of white-listing, but also that Bit9 possesses an astounding data mine concerning the rate of growth of clean software. From an industry perspective it was fascinating to find out that Bit9, one of the sponsors of Robin Bloor's paper 'AVID (Anti-Virus Is Dead)', is a power user of anti-virus software. Bloor's rant, while firmly rooted in marketing does not depict the reality of his sponsor's situation. Despite this, Bit9 may be able to contribute valuable false-positive feedback to the AV community for the benefit of users.

The hot topic of the event was the impending demise of the WildList. As Andrew Lee pointed out, anti-virus testing exists primarily for marketing. Myles Jordan of Microsoft stated that the reason the industry has hung on to the WildList for so long, and will fight to continue doing so, is because WildList testing is easy to pass. In response, VB's own John Hawes posed the question: why, if WildList testing is so easy to pass, do products in each review fail to detect all WildList samples?

Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users. The WildList brought a standard of scientific repeatability and credibility to testers, however if the sentiments of test and research alike are to be acted upon, the WildList will evolve or die.

As if writing a dirge for the WildList, Verizon announced the acquisition of Cybertrust, ultimately the owner of ICSALabs and the WildList. Representatives of ICSALabs were conspicuous by their absence from the event.

While some test organizations make little or no use of the WildList, Virus Bulletin, West Coast Labs and ICSALabs are well advised to work on a plan B sooner rather than later. Speculation on what would be required for a replacement included an automated system that would not rely upon human reporters.

Testers were reminded of the paramount importance of testing malware, rather than the utter garbage prevalent in some collections. Opinions were more diverse when trying to assess what malicious samples are relevant in a test, and whether non-contextual tests against active malware are acceptable. Symantec in particular beat the drum of testing an entire suite holistically as opposed to discrete modules.

Of particular delight to many of us was the opportunity to witness the self-proclaimed swan song of respected AV testing pioneer Prof. Dr Klaus Brunnstein, who concluded the event with a history of malware testing and urged the 'Siamese twins' to go forward in a productive manner that recognizes the symbiosis between the camps. Will this workshop make a difference? Will we see improvements in testing as a result? In a highly imperfect industry that places so much emphasis on perfection, if progress has been made it is unclear whether we will recognize it - but we are, if nothing else, persistent.

Credit is due to AV-Comparatives.org, AV-Test.org, Virus Bulletin, and West Coast Labs for the courage to enter the lion’s den at dinner time!



Latest articles:

VB2017 paper: Browser attack points still abused by banking trojans

With the ever-increasing use of banking-related services on the web, browsers have naturally drawn the attention of malware authors. They are interested in adjusting the behaviour of the browsers for their purposes, namely intercepting the content of…

Does malware based on Spectre exist?

It is likely that, by now, everyone in computer science has at least heard of the Spectre attack. Since many excellent explanations of the attack already exist, this article focuses on the probability of finding Spectre being exploited on Android…

EternalBlue: a prominent threat actor of 2017–2018

At the centre of last year's infamous WannaCry ransomware attack was an NSA exploit leaked by the Shadow Brokers hacker group, known as ‘EternalBlue’. The worm-like functionality of the exploit made a deadly impact by propagating to interconnected…

VB99 paper: Giving the EICAR test file some teeth

There are situations that warrant the use of live viruses. There are also situations where the use of live viruses is unwarranted. Specifically, live viruses should not be used when safer and equally effective methods can be used to obtain the…

Powering the distribution of Tesla stealer with PowerShell and VBA macros

Since their return more than four years ago, Office macros have been one of the most common ways to spread malware. In this paper, Aditya K Sood and Rohit Bansal analyse a campaign in which VBA macros are used to execute PowerShell code, which in…

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.