VBWeb Comparative Review Summer 2017

Martijn Grooten & Adrian Luca

Virus Bulletin

Copyright © Virus Bulletin 2017



When we think of malware spreading via the web, it is tempting to imagine an executable sitting on a rogue (or compromised) server that gets downloaded via a web browser and then run on the local machine. While this scenario does happen, there is a far more subtle way in which malware can make it onto one's device during a browsing session: exploit kits.

Exploit kits have been a threat for several years, and though recently in decline, they remain an infection vector that is not to be ignored. Other than the fact that they often don't require any user interaction, exploit kits often execute malware on the local machine without a malicious executable being downloaded in the browser, thus making detection difficult.

While keeping your software up to date remains the first and most important step you should take to avoid being infected via an exploit kit, it is good to know that security solutions are able to block such activity, even if you (or your employees) are behind schedule on patching. Indeed, this report once again shows that there are solutions that provide excellent protection against today's web threats.


Multiple solutions to the same threat

During August 2017, a number of web security products were run in Virus Bulletin's test lab and exposed to various real-time, web-based threats, including exploit kits and direct malware downloads. Virus Bulletin applies the same rule to all of its tests: each participating vendor must decide prior to the start of the test whether they want the results of the test to be made public, or whether they want to keep the results private, for internal use as quality assurance. In this test two vendors opted to go public with their results, while another four were tested privately.

The products blocked between 90 and 100 per cent of exploit kits, and between 87 and 99 per cent of direct malware downloads. While this shows what a great job products are doing of blocking malware, the details show that there are differences between the products: a web security gateway can help a lot, but some can help a more than others.


The web threat landscape, summer 2017

Though the web threat landscape remains volatile, on the face of it, not much has changed since spring: RIG remains by far the most prevalent exploit kit, with others either having disappeared or having become very localized threats. If you were infected through an exploit kit this year, it was most likely to have been RIG.

RIG uses various campaigns though, each with slightly different characteristics, thus making it far more than a single threat. This was also reflected in the variety of payloads we saw during the test period, which included various kinds of ransomware and other kinds of malware.

RigEK_traffic.pngRIG exploit kit traffic.



Fortinet FortiGate

Drive-by download rate: 100.0%
Malware block rate: 98.6%
Weighted average: 99.9%
Potentially malicious rate: 97.5%


Fortinet's FortiGate appliance blocked all of the more than 400 exploit kits seen in this test, demonstrating that the gateway product continues to provide an excellent first line of defence. The detection rate of direct malware downloads was very good too, with only a handful missed.

Thus both for keeping up with the threat landscape and for the continued protection it offers, Fortinet is well deserving of another VBWeb award and we are pleased to recommend this product to organizations looking to mitigate web-based threats.

fortiguard_detection2.pngFortiGate interface.


Trustwave Secure Web Gateway

Drive-by download rate: 100.0%
Malware block rate: 96.6%
Weighted average: 99.7%
Potentially malicious rate: 99.2%


Exploit kits remain no problem for Trustwave's Secure Web Gateway: once again, not one was missed by the product, while more than 96% of direct malware downloads (where a web gateway is the first line of defence before an endpoint security product will try and block the threat) were blocked too. But besides the excellent performance in this test, it is the product's strong performance over several tests that its developers can be proud of.

As such, Trustwave fully deserves another VBWeb award and we are pleased to recommend this product to organizations looking to mitigate web-based threats.

trustwave_detection1.pngTrustwave interface.


Appendix: The test methodology

The test ran from 27 July to 15 August 2017, during which period we gathered a large number of URLs (most of which were found through public sources) which we had reason to believe could serve a malicious response. We opened the URLs in one of our test browsers, selected at random.

When our systems deemed the response sufficiently likely to fit one of various definitions of 'malicious', we made the same request in the same browser a number of times, each with one of the participating products in front of it. The traffic to the filters was replayed from our cache within seconds of the original request having been made, thus making it a fully real-time test.

We did not need to know at this point whether the response was actually malicious, thus our test didn't depend on malicious sites that were already known to the security community. During a review of the corpus some days later, we analysed the responses and discarded cases for which the traffic was not deemed malicious.

In this test, we checked products against 428 drive-by downloads (exploit kits) and 584 direct malware downloads. To qualify for a VBWeb award, the weighted average catch rate of these two categories, with weights of 90% and 10% respectively, needed to be at least 70%.

We also checked the products against 241 URLs that we deemed 'potentially malicious'. These were URLs for which we had strong evidence that they would serve a malicious response in some cases, but they didn't when we requested it. There could be a number of reasons for this, from server-side randomness to our test lab being detected by anti-analysis tools.

While one can have a perfectly good web security product that doesn't block any of these, we believe that blocking such URLs can serve as an indication of a product's ability to block threats proactively without inspecting the traffic. For some customers this could be important, and for developers this is certainly valuable information, hence we decided to include it in this and future reports.

The test focused on unencrypted HTTP traffic. It did not look at extremely targeted attacks or possible vulnerabilities in the products themselves.


Test machines

Each request was made from a randomly selected virtual machine using one of the available browsers. The machines ran either Windows XP Service Pack 3 Home Edition 2002 or Windows 7 Service Pack 1 Ultimate 2009, and all machines ran slightly out-of-date browsers and browser plug-ins.

Download PDF



Latest reviews:

VBSpam Comparative Review

In this test the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist was measured against various streams of wanted, unwanted and malicious emails.

VBSpam Comparative Review - June 2021

In this test the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist was measured against various streams of wanted, unwanted and malicious emails.

VB100 Certification Report - April 2021

This report details the VB100 certification results of 38 anti-malware products from 33 different vendors tested during March and April 2021.

VBSpam Comparative Review - March 2021

The performance of 28 email security products was measured against various streams of wanted, unwanted and malicious emails. The results for the 10 products that elected to be tested publicly are included in this report.

VB100 Certification Report - February 2021

This report details the VB100 certification results of 43 anti-malware products from 38 different vendors during January and February 2021.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.