VBWeb is a certification and comparative test for web security products, sometimes referred to as web filters. A product that earns the VBWeb certification can be considered to meet a minimum standard of quality when it comes to the blocking of malicious websites and URLs (over HTTP and HTTPS).
Please be advised that while the VBWeb test runs in a lab that mimics a real-life environment relatively well, there may be scenarios that are not fully covered by the test.
The VBWeb test is run as a single continuous test. Participating products are required to filter a mix of test cases from various sources in a number of different categories. Results are reported (both in public reports and privately shared feedback) separately for the different categories.
The VBWeb set-up runs continuously throughout the year, with vendors of participating products receiving regular feedback on their products' performance (this feedback is for the vendors' own internal information and is not made public by either Virus Bulletin or the participating vendors). Four periods are designated as official test periods and, depending on the type of testing agreement in place, products are either tested privately (results are for vendor internal information only) or publicly (results are included in the test report published on Virus Bulletin’s website and certification logos are awarded dependant on the products' performance). Each official test period lasts about two weeks, but because the test uses live URLs, the length may vary depending on the threat landscape.
The public test only includes products belonging to vendors that have committed to the public test. A product whose vendor has committed to the public test may not be withdrawn once the testing starts, unless technical problems prevent successful completion of testing.
Virus Bulletin tests products that have the ability to block malicious websites. This includes locally hosted products that run as either a transparent or an explicit proxy, cloud-based solutions, and URL- or domain-blocklists.
The way a product is set up in our lab is flexible and depends on the type of product. For locally hosted products we use two transparent proxies: one between the client machine and the product and one between the product and the Internet. The former is used to record what is being filtered, the latter is used to replay responses. For cloud-based solutions, only the first kind of proxy exists.
The client machine runs various browsers on the Windows operating system, with both browsers and plug-ins running recent, but not completely up-to-date versions.
A 'case' is defined as anything that results from opening a URL in the browser. A case can be good or bad (though there are various kinds of bad, as well as some cases that we don't consider because they're somewhere in between) and a product can make a right or wrong decision handling it.
During the test, the cases are classified automatically into categories (drive-by downloads / malware / phishing, etc.) and according to what decision the product has made (pass / block). This is followed by a second round of manual checks and classification.
A block is any method that stops the malicious part being delivered to the client:
Within each category the way the product has dealt with each case is counted as either a pass or a block, and a percentage score is calculated.
It should be noted that cloud-based products included in the VBWeb test are scored a little differently. As with the other products hosted in our lab, we replay previously recorded requests through cloud-based products (the requests are played in near-real time), but as we do not control the connection between the product and the Internet, we cannot replay the response. Thus it is possible that a request that results in a malicious response in our test lab results in a non-malicious response when replayed through a cloud-based product. We consider such cases full blocks, as this is the user experience, but because a cloud-based product isn’t always served the malicious content by the exploit kits, for the purpose of calculating block rates we only count these instances with a weight of 0.5 and call this a 'partial block'.
Virus Bulletin relies on various public and private sources, including its own threat intelligence gathering efforts. We also use a private API provided by Active Defense for the exploit kits part.
The following categories are included in the test:
To qualify for a VBWeb award, the weighted average catch rate of drive-by downloads and direct malware download categories, with weights of 90% and 10% respectively, must be at least 80%. In our tests and reports we also include numbers relating to other kinds of malicious traffic, e.g. scams and phishing, but these do not count towards certification.
Regular feedback is provided every two weeks to participants outside official test periods and holidays. Feedback can be contested and can also be used to improve products but, because of the former, should not be assumed to be 100% accurate.
Feedback consists of csv files with the cases included in the test, PCAP files with network traffic recording, and screenshots for phishing cases.
In case of disputes, especially where they concern possible technical issues, log files of both the product and VB’s own systems may be consulted. Virus Bulletin's decision is final, though.
When the feedback concerns an official test, we require vendors to send any disputes within seven days.
The test is paid for by the participating vendors. No other funds are received.