Changes to the VB test sets


Matt Ham

Virus Bulletin, UK
Editor: Helen Martin


VB's product reviewer Matt Ham describes some changes to the test sets used for VB's comparative reviews.


This month's reviewer activity has been concentrated behind the scenes, with some labour-intesive changes being made in the test sets used for VB's comparative reviews.

Clean test set

Over the years VB's clean test sets have consisted of a reasonably representative selection of files. Recently, however, there has been some concern over the inclusion of a large number of dynamically compressed files. In most cases these are installers, which contain multiple executables under potentially proprietary encryption or compression algorithms. While a small number of these would be expected in everyday on-demand scanning, the test sets contain a far larger percentage than one would expect to encounter in a real-world situation.

With the number of scanners that contain routines for delving into such files on the increase, it has become apparent that such in-depth investigation has a severe impact on speed of scanning. Therefore, the inclusion of a large number of installer files in the clean test sets was putting the more thorough products at a disadvantage in terms of the scanning speeds we reported.

To resolve the situation, the clean executables test set has been split into two. One set contains 'pure' executables, while the other contains these dynamically compressed executables. It is hoped that this will enable a better breakdown of any future scanning speed issues.

Spyware test set

The second major change comes in the form of a new test set – a spyware test set. Currently, there are no plans to make the detection of samples in this test set a requirement for achieving a VB 100% award.

While recognising that it is not the most stringent of test methodologies, the current plans are not to look at spyware detection capabilities on machines that are already compromised. Instead, it is envisaged that the files included in the test set will be the initial vector of the malware in question. Thus the downloaded file of a spyware application, Trojanised software with spyware functionality, backdoor servers and the like will make up the bulk of samples in the test set. As the compilation of both the spyware test set and testing methodology is still a work in progress, I would be very pleased to receive comments and suggestions at



Latest articles:

VB2018 paper: Office bugs on the rise

It has never been easier to attack Office vulnerabilities than it is nowadays. In this paper Gabor Szappanos looks more deeply into the dramatic changes that have happened in the past 12 months in the Office exploit scene.

VB2018 paper: Tracking Mirai variants

Mirai, the infamous DDoS botnet family known for its great destructive power, was made open source soon after being found by MalwareMustDie in August 2016, which led to a proliferation of Mirai variant botnets. This paper presents a set of Mirai…

VB2018 paper: Hide’n’Seek: an adaptive peer-to-peer IoT botnet

This paper presents a thorough analysis of the inner workings of Hide’n’Seek, a peer-to-peer IoT botnet discovered in January 2018. With an exploit table that can be updated in memory and modular in its approach, Hide’n’Seek gives us a glimpse of…

Botception: botnet distributes script with bot capabilities

Researchers Jan Sirmer and Adolf Streda describe the branch of the Necurs botnet that they have been monitoring, the changes it has undergone in the course of a year, and present an analysis of the next stage of the attack: Flawed Ammy.

VB2018 paper: Since the hacking of Sony Pictures

Minseok (Jacky) Cha describes various attacks in Korea which occurred after the Sony Pictures hacking incident and which are suspected to be the work of the same group, the Lazarus Group.

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.