VBWeb Comparative Review - Summer 2019

Martijn Grooten & Adrian Luca

Virus Bulletin

Copyright © 2019 Virus Bulletin



Together with email1, the web is one of the two major malware infection vectors through which organizations and individuals get infected with malware. Most organizations use security products to minimize the risk of malware making it onto the network this way, thus avoiding having to rely on security products running on the endpoint.

In the VBWeb tests, which form part of Virus Bulletin's test suite, we measure the performance of web security products against a range of live web threats. We publish quarterly reports on the performance of the products that have opted to be included in our public testing. The reports also include an overview of the current state of the web‑based threat landscape.


The Summer 2019 threat landscape

Recent months have seen some interesting developments when it comes to web-based malware. On the one hand, the shutdown of Coinhive in March 2019 led to a sharp decline in illicit in-browser mining activity, though it will be interesting to see if the recent increase in cryptocurrency values will lead malicious authors to return to mining for such currencies in users' browsers.

At the same time, exploit kits are making some kind of comeback. Just after this test ended, the GreenFlash Sundown kit2 became very active during a large malvertising campaign. During the test itself, we saw hundreds of cases of six different exploit kits: Rig, Spelevo, Fallout, Kaixin, Underminer and the Router exploit kit, which targets vulnerable routers rather than attempting to infect the endpoint.

execution_after_rig.pngInfection attempt following successful execution of the Rig exploit kit.

We saw these exploit kits deliver ransomware such as Sodinokibi and Buran, banking trojans such as IcedID and Kronos, as well as other kinds of malware such as the AZORult information stealer and the Pitou spambot.

We also saw a wide variety of direct malware downloads, including Emotet, GandCrab, Pony, Nanobot and Troldesh. Thankfully, tested products had very few problems blocking malware in any of these categories.

New in this test is the category of phishing websites. Such sites aren't malicious in a strict sense – in that they won't infect a user's computer – but the stealing of credentials is an important cybercriminal activity. Despite the lack of any malicious code making these kinds of sites inherently harder to block, we were pleased to see products block over 95 per cent of them.



It should be noted that one of the products included in this VBWeb test is a cloud-based product. As with the other products hosted in our lab, we replay previously recorded requests through cloud-based products3, but as we do not control the connection between the product and the Internet, we cannot replay the response.

Thus it is possible that a request that results in a malicious response in our test lab results in a non-malicious response when replayed through a cloud-based product. We consider such cases full blocks, as this is the user experience, but because a cloud-based product isn't always served the malicious content by the exploit kits, for the purpose of calculating block rates we only count these instances with a weight of 0.5. However, in the case of the particular cloud-based product included this test, all exploit kits were blocked, meaning that the weighting would not have made a difference.


Fortinet FortiGate

Drive-by download rate 100.0% vbweb-verified.jpg
Malware block rate 99.5%
Phishing block rate 95.2%
Cryptocurrency miner block rate 100.0%
False positive rate 0.0%


Fortinet's FortiGate appliance continued its unbroken run of VBWeb awards that goes back several years, blocking all drive-by download cases and almost all direct malware downloads. With over 95 per cent of phishing sites blocked, this kind of malicious site doesn't pose a big problem for FortiGate either.

fortigate_18june_02.pngFortinet FortiGate.



Drive-by download rate 100.0% vbweb-verified.jpg
Malware block rate 99.3%
Phishing block rate 96.9%
Cryptocurrency miner block rate 100.0%
False positive rate 0.5%


iBoss showed that its impressive performance in its debut test was no exception, blocking all drive-by download cases (exploit kits) in this test, as well as all directly downloaded malware. iBoss also blocked almost 97 per cent of phishing sites. The product's second VBWeb certification is thus well deserved.



Appendix: The test methodology

The test ran from 7 June 2019 to 24 June 2019 during which period we gathered a large number of URLs (most of which were found through public sources) which we had reason to believe could serve a malicious response. We opened the URLs in one of our test browsers, selected at random.

When our systems deemed the response sufficiently likely to fit one of various definitions of 'malicious', we made the same request in the same browser a number of times, each with one of the participating products in front of it. The traffic to the filters was replayed from our cache within seconds of the original request having been made, thus making it a fully real-time test.

We did not need to know at this point whether the response was actually malicious, thus our test didn't depend on malicious sites that were already known to the security community. During a review of the test corpus some days later, we analysed the responses and discarded cases for which the traffic was not deemed malicious.

In this test, we checked products against 224 drive‑by downloads (exploit kits), 1,338 direct malware downloads and 547 phishing sites, a category which also includes sites that trick the user into calling a phone number. To qualify for a VBWeb award, the weighted average catch rate of the first two of these categories, with weights of 90% and 10% respectively, needed to be at least 80%.

The 'potentially malicious' cases that were included in previous reports have been removed from the public test reports.

The test focused on both HTTP and HTTPS traffic. It did not look at extremely targeted attacks or possible vulnerabilities in the products themselves.


Test Machines

Each request was made from a randomly selected virtual machine using one of the available browsers. The machines ran either Windows XP Service Pack 3 Home Edition 2002, or Windows 7 Service Pack 1 Ultimate 2009 and all ran slightly out-of-date browsers and browser plug-ins.



1 See the regular VBSpam reports on the email-based threat landscape and email security products' ability to protect email accounts:

2 https://blog.malwarebytes.com/threat-analysis/2019/06/greenflash-sundown-exploit-kit-expands-via-large-malvertising-campaign/.

3 The requests are replayed in near real time.

Download PDF



Latest reviews:

VB100 Certification Report - April 2020

This report details the VB100 certification results of 42 anti-malware products from 36 different vendors during March and April 2020.

VBWeb Comparative Review - Spring 2020

In the final VBWeb test we measure the performance of three web security products against a wide range of live web threats, including phishing threats. This report also includes an overview of the Spring 2020 threat landscape.

VBSpam Comparative Review - March 2020

In the March 2020 VBSpam test 10 of the 11 participating full solutions obtained a VBSpam award and seven of them performed well enough to earn a VBSpam+ award.

VB100 Certification Report - February 2020

This report details the VB100 certification results of 40 anti-malware products from 36 different vendors during January and February 2020.

VBWeb Comparative Review - Winter 2020

The VBWeb tests measure the performance of web security products against a wide range of live web threats, also including phishing threats. This report also includes an overview of the Winter 2019/2020 threat landscape.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.