VBWeb Methodology

The rules of the VBWeb comparative test

What is the purpose of the VBWeb tests?

In its bi-monthly VBWeb tests, Virus Bulletin will test the ability of network-based solutions to prevent malicious websites from serving end-users the payload (typically a malicious file).

When will the tests commence?

The first test started on Friday 4 December 2015. The results of this test can be read here. Hereafter, tests will be run every two months.

What is the difference between a private and a public test?

Products submitted to either kind of test will be tested under exactly the same conditions. The only differences are that the results for a privately tested product will not be published (either by Virus Bulletin or by the vendor) and the product will not be eligible for the VBWeb certification logo.

For reasons of transparency we would prefer to see vendors submit to public testing, so we charge lower fees for participating publicly.

How do the tests work?

Live malicious traffic is requested by VB’s test network and the responses are stored in a local cache. They are subsequently (and immediately) replayed to the various products. Assuming the original, unfiltered request resulted in a malicious payload being served, a product is considered to have blocked the request if the payload is not served after it filtered the traffic.

What type of traffic will the tests include?

Our focus is on URLs (or ‘cases’; see below) that result in malware being downloaded, either through a direct download or via drive-by downloads. We will likely include other kinds of malicious traffic, e.g. scams and phishing, but these won’t count towards certification.

What is required to achieve certification?

The certification requirements will be decided based on standards achieved in trials and initial tests, and will be detailed in each test report.

Will the tests look at false positives too?

We will regularly visit legitimate, non-malicious sites to verify they aren’t being blocked. In the future, we may be able to add a more rigorous false positive test.

Will tests include the blocking of malicious HTTPS sites?

We are working on this at the moment. For now, the traffic will be plain HTTP only.

Will the products be connected to the Internet?

Yes, products will have full Internet connectivity throughout the test.

Will solutions that run on the desktop itself be tested?

We are looking at ways to do this. For now, our focus is on products that run on the gateway as an explicit or transparent proxy.

Will solutions that are hosted in the cloud be tested?

In principle we intend to include such products, but there are some potential issues here. Please contact us to discuss how your particular product is set up.

Will VB provide detailed feedback on missed cases?

We will provide PCAPs of the network traffic.

What is the difference between a 'URL' and a 'case'?

A URL is a text string, such as http://www.example.com/foo?id=bar

When a URL is entered into a browser, an HTTP request is made to the web server. The response sent by the web server usually results in the browser making further requests: images, CSS libraries, etc. A 'case' is anything that happens as a consequence of the URL being entered, that is the total collection of requests and responses that are generated.

Each case starts with a URL being entered into the browser, but the responses usually don't just depend on this URL. A different browser, a different IP address, or a different time can also result in different responses and this is especially an issue with malicious websites. For that reason, we don't refer to 'bad URLs' but instead to 'bad cases’. The only way to find out whether a case is malicious is by actually making the request.

So one initial URL results in many URLs being requested - the total collection of these requests and responses are the resulting case. A case can be 'good' or 'bad' (or something in between).


Latest Report

The latest VBWeb comparative test report

VBWeb Test Schedule

The schedule for upcoming VBWeb test reports

VBWeb Methodology

The rules of the VBWeb comparative test

VBWeb Test Archive

Archive of VBWeb test reports.

VB Testing