VBWeb Comparative Review Summer 2018

Martijn Grooten & Adrian Luca

Virus Bulletin

Copyright © Virus Bulletin 2018



Together with email1, web is one of the major vectors through which organizations and individuals get infected with malware. Despite a decline in activity, exploit kits remain the most prominent web-based threat and are a reminder that browsers and browser plug-ins should take absolute priority when it comes to patching: exploit kits almost always exploit vulnerabilities for which a patch has been made available.

But because people do make mistakes, and organizations find it hard always to patch software right away, web security products provide an important extra layer of defence.


The May 2018 threat landscape

Two kits continue to dominate the global exploit kit landscape: Rig and Grandsoft, both of which are served via a number of different campaigns. A single campaign often serves different kinds of malware to different victims, and though cryptocurrency miners remain popular among cybercriminals, we continue to see more traditional and more serious threats, such as banking trojans and ransomware, being served.

Of course, cryptocurrency mining is also performed via malicious and/or compromised websites. We do not make the blocking of such threats a requirement to achieve VBWeb certification as there are legitimate sites that perform this kind of activity; nevertheless, we were pleased to find that all solutions in our test (including those being tested privately) blocked 100% of cryptocurrency miners.

As in the previous report, we continue to see threats being delivered via HTTPS. Though the privacy implications of an organization intercepting HTTPS for its employees should be well understood, it is important to note that not doing so would result in a not insignificant amount of malicious web traffic bypassing the security product in place.



Fortinet FortiGate

Drive-by download rate
100.0% vbweb-verified.jpg
Malware block rate 98.7%
Weighted average 99.9%
Potentially malicious rate 97.9%
Cryptocurrency miner block rate 100.0%
False positive rate

In yet another test, Fortinet’s FortiGate appliance blocked every one of the hundreds of drive-by downloads it was served, showing continued excellence when it comes to blocking this type of threat.

With a weighted average block rate of 99.9%, we are pleased to award Fortinet its eighth VBWeb award.


FortiGate interface.


Trustwave Secure Web Gateway

Drive-by download rate
100.0% vbweb-verified.jpg
Malware block rate 98.9%
Weighted average 99.9%
Potentially malicious rate 93.8%
Cryptocurrency miner block rate 100.0%
False positive rate

Trustwave’s Secure Web Gateway continues its excellent performance on our test bed, once again blocking all exploit kits – no trivial achievement given how such kits constantly evolve in an attempt to stay one step ahead of the security vendors.

With a weighted average block rate of 99.9%, yet another VBWeb award is well deserved by Trustwave.

trustwave_25_may_2018.png Trustwave Secure Web Gateway interface.


Appendix 1: The Test Methodology

The test ran from 16 to 31 May 2018, during which period we gathered a large number of URLs (most of which were found through public sources) which we had reason to believe could serve a malicious response. We opened the URLs in one of our test browsers, selected at random.

When our systems deemed the response sufficiently likely to fit one of various definitions of ‘malicious’, we made the same request in the same browser a number of times, each with one of the participating products in front of it. The traffic to the filters was replayed from our cache within seconds of the original request having been made, thus making it a fully real-time test.

We did not need to know at this point whether the response was actually malicious, thus our test didn’t depend on malicious sites that were already known to the security community. During a review of the test corpus some days later, we analysed the responses and discarded cases for which the traffic was not deemed malicious.

In this test, we checked products against 291 drive-by downloads (exploit kits) and 1419 direct malware downloads. To qualify for a VBWeb award, the weighted average catch rate of these two categories, with weights of 90% and 10% respectively, needed to be at least 70%.

We also checked the products against 243 URLs that we deemed ‘potentially malicious’. These were URLs for which we had strong evidence that they would serve a malicious response in some cases, but they didn’t when we requested it. There could be a number of reasons for this, from server-side randomness to our test lab being detected by anti-analysis tools.

While one can have a perfectly good web security product that doesn’t block any of these, we believe that blocking such URLs can serve as an indication of a product’s ability to block threats proactively without inspecting the traffic. For some customers this could be important, and for developers this is certainly valuable information, hence we decided to include it in this and future reports.

The test focused on unencrypted HTTP traffic. It did not look at extremely targeted attacks or possible vulnerabilities in the products themselves.


Appendix 2: Test Machines

Each request was made from a randomly selected virtual machine using one of the available browsers. The machines ran either Windows XP Service Pack 3 Home Edition 2002, or Windows 7 Service Pack 1 Ultimate 2009 and all ran slightly out-of-date browsers and browser plug-ins.



1 See the regular VBSpam reports on the email-based threat landscape and email security products’ ability to protect email accounts: https://www2.virusbtn.com/testing/vbspam/.

Download PDF



Latest reviews:

VB100 Certification Report - April 2020

This report details the VB100 certification results of 42 anti-malware products from 36 different vendors during March and April 2020.

VBWeb Comparative Review - Spring 2020

In the final VBWeb test we measure the performance of three web security products against a wide range of live web threats, including phishing threats. This report also includes an overview of the Spring 2020 threat landscape.

VBSpam Comparative Review - March 2020

In the March 2020 VBSpam test 10 of the 11 participating full solutions obtained a VBSpam award and seven of them performed well enough to earn a VBSpam+ award.

VB100 Certification Report - February 2020

This report details the VB100 certification results of 40 anti-malware products from 36 different vendors during January and February 2020.

VBWeb Comparative Review - Winter 2020

The VBWeb tests measure the performance of web security products against a wide range of live web threats, also including phishing threats. This report also includes an overview of the Winter 2019/2020 threat landscape.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.