Changes to the VB test sets


Matt Ham

Virus Bulletin, UK
Editor: Helen Martin


VB's product reviewer Matt Ham describes some changes to the test sets used for VB's comparative reviews.


This month's reviewer activity has been concentrated behind the scenes, with some labour-intesive changes being made in the test sets used for VB's comparative reviews.

Clean test set

Over the years VB's clean test sets have consisted of a reasonably representative selection of files. Recently, however, there has been some concern over the inclusion of a large number of dynamically compressed files. In most cases these are installers, which contain multiple executables under potentially proprietary encryption or compression algorithms. While a small number of these would be expected in everyday on-demand scanning, the test sets contain a far larger percentage than one would expect to encounter in a real-world situation.

With the number of scanners that contain routines for delving into such files on the increase, it has become apparent that such in-depth investigation has a severe impact on speed of scanning. Therefore, the inclusion of a large number of installer files in the clean test sets was putting the more thorough products at a disadvantage in terms of the scanning speeds we reported.

To resolve the situation, the clean executables test set has been split into two. One set contains 'pure' executables, while the other contains these dynamically compressed executables. It is hoped that this will enable a better breakdown of any future scanning speed issues.

Spyware test set

The second major change comes in the form of a new test set – a spyware test set. Currently, there are no plans to make the detection of samples in this test set a requirement for achieving a VB 100% award.

While recognising that it is not the most stringent of test methodologies, the current plans are not to look at spyware detection capabilities on machines that are already compromised. Instead, it is envisaged that the files included in the test set will be the initial vector of the malware in question. Thus the downloaded file of a spyware application, Trojanised software with spyware functionality, backdoor servers and the like will make up the bulk of samples in the test set. As the compilation of both the spyware test set and testing methodology is still a work in progress, I would be very pleased to receive comments and suggestions at



Latest articles:

VB2017 paper: The life story of an IPT – Inept Persistent Threat actor

This paper describes the ability of an amateur attacker with no technical skills to achieve success in his criminal enterprise. We will follow a Polish threat actor, known as ‘Thomas’, in his career as a wannabe cybercriminal from late 2011 until…

VB2017 paper: The router of all evil: more than just default passwords and silly scripts

In the last couple of years, we have seen a few highly sophisticated router attacks and pieces of malware. This paper looks at two case studies: the Netgear router attack involving the Multiple Netgear Routers Remote Command Injection Vulnerability…

A review of the evolution of Andromeda over the years before we say goodbye

Andromeda, also known as Gamaru and Wauchos, is a modular and HTTP-based botnet that was discovered in late 2011. From that point on, it managed to survive and continue hardening by evolving in different ways. This paper describes the evolution of…

VB2012 paper: Malware taking a bit(coin) more than we bargained for

When a new system of currency gains acceptance and widespread adoption in a computer-mediated population, it is only a matter of time before malware authors attempt to exploit it. As of halfway through 2011, we started seeing another means of…

VB2017 paper: VirusTotal tips, tricks and myths

Outside of the anti-malware industry, users of VirusTotal generally believe it is simply a virus-scanning service. Most users quickly reach erroneous conclusions about the meaning of various scanning results. At the same time, many very technical…

Bulletin Archive