VBWeb Comparative Review - Winter 2019

Martijn Grooten & Adrian Luca

Virus Bulletin

Copyright © 2019 Virus Bulletin



Together with email1, the web is one of the two major infection vectors through which organizations and individuals get infected with malware. Most organizations use web security products to minimize the risk of malware making it onto the network this way, thus avoiding having to rely solely on security products that run on the endpoint.

In the VBWeb tests we measure the performance of web security products against a wide range of live web threats. We publish quarterly reports on the performance of the products that have opted to be included in our public testing. The reports also include an overview of the current state of the web‑based threat landscape.


The Winter 2019 threat landscape

Though cybercrime is a global phenomenon, a significant number of malware campaigns are run by individuals who take a break during the Russian Orthodox Christmas holidays. Indeed, we saw a decline in activity during this period, with many exploit kits either reducing or stopping their activity altogether.

We believe that a number of exploit kits continued to be active during this period, but we only observed two, Rig and Fallout, during our test, with Fallout not delivering a full payload. This means that all 277 of the exploit kits observed were part of Rig, which uses a number of different gates.

The Gandcrab ransomware continues to be a payload commonly delivered by exploit kits, while we also saw infostealers like AZORult and Vidar being delivered, the latter of which is fairly new2, as well as some adware families. We observed at least one case where a download of AZORult was followed by a download of the infamous Emotet malware.

RigEK_post_infection_traffic.pngPost-infection traffic following a successful infection attempt by the Rig exploit kit serving first AZORult and then Emotet (traffic highlighted with EKFiddle in Fiddler).

Emotet was also seen regularly as the payload of direct malware downloads, which is not surprising given that it is often served via links in spam emails. Other payloads we observed included Ramnit, Hancitor and Mirai. We should note that it is likely that some of these malware samples would have gone on to install further payloads in a real‑world scenario.

As we have seen in the past, the products in our test performed extremely well against drive-by downloads and the related, but less severe threat of in-browser cryptocurrency mining, with both of the products in this public report blocking 100% of the cases in both categories. Block rates of direct malware downloads were not quite as impressive, but still very high.



Fortinet FortiGate

Drive-by download rate
100.0% vbweb-verified.jpg
Malware block rate 98.5%
Weighted average 99.9%
Potentially malicious rate 99.3%
Cryptocurrency miner block rate 100.0%
False positive rate

A long test period didn't pose any problems for Fortinet's FortiGate appliance, which blocked all drive‑by downloads and all instances of in-browser mining we observed in our test. It also blocked all but a handful of direct malware downloads, meaning that Fortinet easily achieves its eighth VBWeb certification.

fortiguard_18_dec_2018_1.pngFortinet FortiGate.


Trustwave Secure Web Gateway

Drive-by download rate
100.0% vbweb-verified.jpg
Malware block rate 96.5%
Weighted average 99.7%
Potentially malicious rate 88.9%
Cryptocurrency miner block rate 100.0%
False positive rate


The last test saw Trustwave's web security product catch up quickly with new threats, so we weren't surprised to see it block all drive-by downloads and all instances of in-browser mining in this test. A small number of direct malware downloads slipped through, but in practice such misses would be mitigated by security products running on the endpoint. Trustwave is thus fully deserving of its seventh VBWeb certification.

trustwave_18_dec_2018_4.pngTrustwave Secure Web Gateway.


Appendix: The test methodology

The test ran from 26 November 2018 to 13 January 2019 (with a short break taken around the holiday season), during which period we gathered a large number of URLs (most of which were found through public sources) which we had reason to believe could serve a malicious response. We opened the URLs in one of our test browsers, selected at random.

When our systems deemed the response sufficiently likely to fit one of various definitions of 'malicious', we made the same request in the same browser a number of times, each with one of the participating products in front of it. The traffic to the filters was replayed from our cache within seconds of the original request having been made, thus making it a fully real-time test.

We did not need to know at this point whether the response was actually malicious, thus our test didn't depend on malicious sites that were already known to the security community. During a review of the test corpus some days later, we analysed the responses and discarded cases for which the traffic was not deemed malicious.

In this test, we checked products against 277 drive‑by downloads (exploit kits) and 1,067 direct malware downloads. To qualify for a VBWeb award, the weighted average catch rate of these two categories, with weights of 90% and 10% respectively, needed to be at least 70%.

We also checked the products against 289 URLs that we call 'potentially malicious'. These were URLs for which we had strong evidence that they would serve a malicious response in some cases, but they didn't when we requested it. There could be a number of reasons for this, from server-side randomness to our test lab being detected by anti-analysis tools.

While one can have a perfectly good web security product that doesn't block any of these, we believe that blocking such URLs can serve as an indication of a product's ability to block threats proactively without inspecting the traffic. For some customers this could be important, and for developers this is certainly valuable information, hence we include it in this report and future reports.

The test focused on unencrypted HTTP traffic. It did not look at extremely targeted attacks or possible vulnerabilities in the products themselves.


Test machines

Each request was made from a randomly selected virtual machine using one of the available browsers. The machines ran either Windows XP Service Pack 3 Home Edition 2002 or Windows 7 Service Pack 1 Ultimate 2009, and all ran slightly out-of-date browsers and browser plug-ins.



1 See the regular VBSpam reports on the email-based threat landscape and email security products’ ability to protect email accounts: https://www.virusbulletin.com/testing/vbspam/.

2 https://fumik0.com/2018/12/24/lets-dig-into-vidar-an-arkei-copycat-forked-stealer-in-depth-analysis/.

Download PDF



Latest reviews:

VBWeb Comparative Review - Winter 2020

The VBWeb tests measure the performance of web security products against a wide range of live web threats, also including phishing threats. This report also includes an overview of the Winter 2019/2020 threat landscape.

VB100 certification report - December 2019

This report details the VB100 certification results of 39 anti-malware products from 33 different vendors during November and December 2019.

VBSpam comparative review - December 2019

In this test – which forms part of Virus Bulletin’s continuously running security product test suite – 11 full email security solutions and five blacklists of various kinds were assembled on the test bench to measure their performance against various…

VB100 Certification Report - October 2019

The October 2019 report details the VB100 certification of 38 anti‑malware products from 33 different vendors tested during September and October 2019.

VBWeb Comparative Review - Autumn 2019

The VBWeb tests measure the performance of web security products against a wide range of live web threats, this time also including phishing threats.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.