AV industry commments on anti-malware testing


Randy Abrams

Eset, USA
Editor: Helen Martin


'Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users' Randy Abrams, Eset.

In the fine tradition of the pioneers of the anti-virus industry, the 1st International Antivirus Testing Workshop was conceived and held in Reykjavik, Iceland last month. Michael St. Neitzel, formerly of Microsoft and Eset, now working for Frisk, had seen one too many unacceptably bad AV tests and decided it was time to bring AV researchers and testers together to try to improve the state of malware testing.

Researchers have beaten up on testers for years with little discernable result, so the notion of such a meeting to improve the status quo may seem a little quixotic unless one realizes that the opponent the AV industry faces is not a windmill, but rather, in the words of Dr Klaus Brunnstein, its Siamese twin.

The presentations were interesting and can be found at http://www.f-prot.com/workshop2007/, however the majority of value came in the discussions that followed the presentations.

A presentation modestly entitled 'Building & leveraging white database for antivirus testing' by Mario Vuksan from Bit9 was the sleeper. The presentation exposed not only the complexities of white-listing, but also that Bit9 possesses an astounding data mine concerning the rate of growth of clean software. From an industry perspective it was fascinating to find out that Bit9, one of the sponsors of Robin Bloor's paper 'AVID (Anti-Virus Is Dead)', is a power user of anti-virus software. Bloor's rant, while firmly rooted in marketing does not depict the reality of his sponsor's situation. Despite this, Bit9 may be able to contribute valuable false-positive feedback to the AV community for the benefit of users.

The hot topic of the event was the impending demise of the WildList. As Andrew Lee pointed out, anti-virus testing exists primarily for marketing. Myles Jordan of Microsoft stated that the reason the industry has hung on to the WildList for so long, and will fight to continue doing so, is because WildList testing is easy to pass. In response, VB's own John Hawes posed the question: why, if WildList testing is so easy to pass, do products in each review fail to detect all WildList samples?

Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users. The WildList brought a standard of scientific repeatability and credibility to testers, however if the sentiments of test and research alike are to be acted upon, the WildList will evolve or die.

As if writing a dirge for the WildList, Verizon announced the acquisition of Cybertrust, ultimately the owner of ICSALabs and the WildList. Representatives of ICSALabs were conspicuous by their absence from the event.

While some test organizations make little or no use of the WildList, Virus Bulletin, West Coast Labs and ICSALabs are well advised to work on a plan B sooner rather than later. Speculation on what would be required for a replacement included an automated system that would not rely upon human reporters.

Testers were reminded of the paramount importance of testing malware, rather than the utter garbage prevalent in some collections. Opinions were more diverse when trying to assess what malicious samples are relevant in a test, and whether non-contextual tests against active malware are acceptable. Symantec in particular beat the drum of testing an entire suite holistically as opposed to discrete modules.

Of particular delight to many of us was the opportunity to witness the self-proclaimed swan song of respected AV testing pioneer Prof. Dr Klaus Brunnstein, who concluded the event with a history of malware testing and urged the 'Siamese twins' to go forward in a productive manner that recognizes the symbiosis between the camps. Will this workshop make a difference? Will we see improvements in testing as a result? In a highly imperfect industry that places so much emphasis on perfection, if progress has been made it is unclear whether we will recognize it - but we are, if nothing else, persistent.

Credit is due to AV-Comparatives.org, AV-Test.org, Virus Bulletin, and West Coast Labs for the courage to enter the lion’s den at dinner time!



Latest articles:

Nexus Android banking botnet – compromising C&C panels and dissecting mobile AppInjects

Aditya Sood & Rohit Bansal provide details of a security vulnerability in the Nexus Android botnet C&C panel that was exploited to compromise the C&C panel in order to gather threat intelligence, and present a model of mobile AppInjects.

Cryptojacking on the fly: TeamTNT using NVIDIA drivers to mine cryptocurrency

TeamTNT is known for attacking insecure and vulnerable Kubernetes deployments in order to infiltrate organizations’ dedicated environments and transform them into attack launchpads. In this article Aditya Sood presents a new module introduced by…

Collector-stealer: a Russian origin credential and information extractor

Collector-stealer, a piece of malware of Russian origin, is heavily used on the Internet to exfiltrate sensitive data from end-user systems and store it in its C&C panels. In this article, researchers Aditya K Sood and Rohit Chaturvedi present a 360…

Fighting Fire with Fire

In 1989, Joe Wells encountered his first virus: Jerusalem. He disassembled the virus, and from that moment onward, was intrigued by the properties of these small pieces of self-replicating code. Joe Wells was an expert on computer viruses, was partly…

Run your malicious VBA macros anywhere!

Kurt Natvig wanted to understand whether it’s possible to recompile VBA macros to another language, which could then easily be ‘run’ on any gateway, thus revealing a sample’s true nature in a safe manner. In this article he explains how he recompiled…

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.