AV industry commments on anti-malware testing

2007-06-01

Randy Abrams

Eset, USA
Editor: Helen Martin

Abstract

'Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users' Randy Abrams, Eset.


In the fine tradition of the pioneers of the anti-virus industry, the 1st International Antivirus Testing Workshop was conceived and held in Reykjavik, Iceland last month. Michael St. Neitzel, formerly of Microsoft and Eset, now working for Frisk, had seen one too many unacceptably bad AV tests and decided it was time to bring AV researchers and testers together to try to improve the state of malware testing.

Researchers have beaten up on testers for years with little discernable result, so the notion of such a meeting to improve the status quo may seem a little quixotic unless one realizes that the opponent the AV industry faces is not a windmill, but rather, in the words of Dr Klaus Brunnstein, its Siamese twin.

The presentations were interesting and can be found at http://www.f-prot.com/workshop2007/, however the majority of value came in the discussions that followed the presentations.

A presentation modestly entitled 'Building & leveraging white database for antivirus testing' by Mario Vuksan from Bit9 was the sleeper. The presentation exposed not only the complexities of white-listing, but also that Bit9 possesses an astounding data mine concerning the rate of growth of clean software. From an industry perspective it was fascinating to find out that Bit9, one of the sponsors of Robin Bloor's paper 'AVID (Anti-Virus Is Dead)', is a power user of anti-virus software. Bloor's rant, while firmly rooted in marketing does not depict the reality of his sponsor's situation. Despite this, Bit9 may be able to contribute valuable false-positive feedback to the AV community for the benefit of users.

The hot topic of the event was the impending demise of the WildList. As Andrew Lee pointed out, anti-virus testing exists primarily for marketing. Myles Jordan of Microsoft stated that the reason the industry has hung on to the WildList for so long, and will fight to continue doing so, is because WildList testing is easy to pass. In response, VB's own John Hawes posed the question: why, if WildList testing is so easy to pass, do products in each review fail to detect all WildList samples?

Agreement was virtually unanimous that the WildList is no longer useful as a metric of the ability of a product to protect users. The WildList brought a standard of scientific repeatability and credibility to testers, however if the sentiments of test and research alike are to be acted upon, the WildList will evolve or die.

As if writing a dirge for the WildList, Verizon announced the acquisition of Cybertrust, ultimately the owner of ICSALabs and the WildList. Representatives of ICSALabs were conspicuous by their absence from the event.

While some test organizations make little or no use of the WildList, Virus Bulletin, West Coast Labs and ICSALabs are well advised to work on a plan B sooner rather than later. Speculation on what would be required for a replacement included an automated system that would not rely upon human reporters.

Testers were reminded of the paramount importance of testing malware, rather than the utter garbage prevalent in some collections. Opinions were more diverse when trying to assess what malicious samples are relevant in a test, and whether non-contextual tests against active malware are acceptable. Symantec in particular beat the drum of testing an entire suite holistically as opposed to discrete modules.

Of particular delight to many of us was the opportunity to witness the self-proclaimed swan song of respected AV testing pioneer Prof. Dr Klaus Brunnstein, who concluded the event with a history of malware testing and urged the 'Siamese twins' to go forward in a productive manner that recognizes the symbiosis between the camps. Will this workshop make a difference? Will we see improvements in testing as a result? In a highly imperfect industry that places so much emphasis on perfection, if progress has been made it is unclear whether we will recognize it - but we are, if nothing else, persistent.

Credit is due to AV-Comparatives.org, AV-Test.org, Virus Bulletin, and West Coast Labs for the courage to enter the lion’s den at dinner time!

twitter.png
fb.png
linkedin.png
googleplus.png
reddit.png

 

Latest articles:

Powering the distribution of Tesla stealer with PowerShell and VBA macros

Since their return more than four years ago, Office macros have been one of the most common ways to spread malware. In this paper, Aditya K Sood and Rohit Bansal analyse a campaign in which VBA macros are used to execute PowerShell code, which in…

VB2017 paper: Android reverse engineering tools: not the usual suspects

In the Android security field, all reverse engineers will probably have used some of the most well-known analysis tools such as apktool, smali, baksmali, dex2jar, etc. These tools are indeed must‑haves for Android application analysis. However, there…

VB2017 paper: Exploring the virtual worlds of advergaming

As adverts in gaming (‘advergaming’) ecosystems continue to become more sophisticated, so the potential complications grow for parents, children and gamers, who just want to play without having to worry about where their data is going (and how it is…

Distinguishing between malicious app collusion and benign app collaboration: a machine-learning approach

Two or more mobile apps, viewed independently, may not appear to be malicious - but in combination, they could become harmful by exchanging information with one another and by performing malicious activities together. In this paper we look at how…

VB2016 paper: Wild Android collusions

Mobile operating systems support multiple communication methods between apps. Unfortunately, these handy inter-app communication mechanisms also make it possible to carry out harmful actions in a collaborative fashion. Two or more mobile apps, viewed…


Bulletin Archive