Changes to the VB test sets


Matt Ham

Virus Bulletin, UK
Editor: Helen Martin


VB's product reviewer Matt Ham describes some changes to the test sets used for VB's comparative reviews.


This month's reviewer activity has been concentrated behind the scenes, with some labour-intesive changes being made in the test sets used for VB's comparative reviews.

Clean test set

Over the years VB's clean test sets have consisted of a reasonably representative selection of files. Recently, however, there has been some concern over the inclusion of a large number of dynamically compressed files. In most cases these are installers, which contain multiple executables under potentially proprietary encryption or compression algorithms. While a small number of these would be expected in everyday on-demand scanning, the test sets contain a far larger percentage than one would expect to encounter in a real-world situation.

With the number of scanners that contain routines for delving into such files on the increase, it has become apparent that such in-depth investigation has a severe impact on speed of scanning. Therefore, the inclusion of a large number of installer files in the clean test sets was putting the more thorough products at a disadvantage in terms of the scanning speeds we reported.

To resolve the situation, the clean executables test set has been split into two. One set contains 'pure' executables, while the other contains these dynamically compressed executables. It is hoped that this will enable a better breakdown of any future scanning speed issues.

Spyware test set

The second major change comes in the form of a new test set – a spyware test set. Currently, there are no plans to make the detection of samples in this test set a requirement for achieving a VB 100% award.

While recognising that it is not the most stringent of test methodologies, the current plans are not to look at spyware detection capabilities on machines that are already compromised. Instead, it is envisaged that the files included in the test set will be the initial vector of the malware in question. Thus the downloaded file of a spyware application, Trojanised software with spyware functionality, backdoor servers and the like will make up the bulk of samples in the test set. As the compilation of both the spyware test set and testing methodology is still a work in progress, I would be very pleased to receive comments and suggestions at



Latest articles:

VB2017 paper: Browser attack points still abused by banking trojans

With the ever-increasing use of banking-related services on the web, browsers have naturally drawn the attention of malware authors. They are interested in adjusting the behaviour of the browsers for their purposes, namely intercepting the content of…

Does malware based on Spectre exist?

It is likely that, by now, everyone in computer science has at least heard of the Spectre attack. Since many excellent explanations of the attack already exist, this article focuses on the probability of finding Spectre being exploited on Android…

EternalBlue: a prominent threat actor of 2017–2018

At the centre of last year's infamous WannaCry ransomware attack was an NSA exploit leaked by the Shadow Brokers hacker group, known as ‘EternalBlue’. The worm-like functionality of the exploit made a deadly impact by propagating to interconnected…

VB99 paper: Giving the EICAR test file some teeth

There are situations that warrant the use of live viruses. There are also situations where the use of live viruses is unwarranted. Specifically, live viruses should not be used when safer and equally effective methods can be used to obtain the…

Powering the distribution of Tesla stealer with PowerShell and VBA macros

Since their return more than four years ago, Office macros have been one of the most common ways to spread malware. In this paper, Aditya K Sood and Rohit Bansal analyse a campaign in which VBA macros are used to execute PowerShell code, which in…

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.