VBWeb Comparative Review - Spring 2020

Adrian Luca

Virus Bulletin

Copyright © 2020 Virus Bulletin



Together with email1 the web is one of the two major malware infection vectors through which organizations and individuals get infected with malware. For this reason most organizations use security products to minimize the risk of malware making it onto the network in this way, thus avoiding having to rely on security products running on endpoints.

In the VBWeb tests, which form part of Virus Bulletin’s test suite, we measure the performance of web security products against a range of live web threats. We have, until now, been publishing quarterly reports on the performance of the products that have opted to be included in our public testing as well as providing an overview of the current state of the web‑based threat landscape. Regrettably, the Spring 2020 edition will be the last such report and the VBWeb test is indefinitely suspended.


The Spring 2020 web threat landscape

Once again, the most active exploit kit we saw during the test period was RIG, which we typically caught through malvertising. A total of at least eight exploit kits are still active, and during the test itself we saw more than 600 cases of six different exploit kits: RIG, Fallout, Spelevo, Underminer, Lewd and Bottle.

The most recent kit, BottleEK2, installs a new banking trojan, named Cinobi3, that targets users in Japan. Other malware we saw downloaded by the various exploit kits included: Dridex, Phorpiex, Raccoon, XMRig, Sodinokibi, Smokeloader, Zloader and Hidenbee.

VBWeb-spring20-fig1.pngBottle Ek traffic.

We also saw almost 500 instances of malware downloads from around 80 families including Emotet, Pony, Troldesh, Ursnif, Azorult and FormBook. Fortunately, the tested products had very few problems blocking malware in any of these categories.

And as was the case in the Winter 2020 VBWeb test4, products also had few problems blocking phishing pages.



It should be noted that one of the products included in this VBWeb test is a cloud-based product. As with the other products hosted in our lab, we replay previously recorded requests through cloud-based products5, but as we do not control the connection between the product and the Internet, we cannot replay the response.

Thus it is possible that a request that results in a malicious response in our test lab results in a non-malicious response when replayed through a cloud-based product. We consider such cases full blocks, as this is the user experience, but because a cloud-based product isn’t always served the malicious content by the exploit kits, for the purpose of calculating block rates we only count these instances with a weight of 0.5. However, in the case of the particular cloud-based product included this test, all exploit kits were blocked, meaning that the weighting would not have made a difference.


Fortinet FortiGate

Drive-by download rate 100.0% vbweb-verified.jpg
Malware block rate 99.4% 
Phishing block rate 95.4% 
False positive rate 0.0%


Fortinet’s FortiGate appliance extends its unbroken run of VBWeb awards going several years, by blocking all drive-by download cases and missing only three of almost 500 direct malware downloads. With over 95 per cent of phishing sites blocked, this kind of malicious site isn’t a big problem for FortiGate either. As such, Fortinet FortiGate fully deserves another VBWeb award. But besides the product's excellent performance in this test, it is its strong performance over 15 tests that its developers can be truly proud of.

FortiGate-VBWeb-Spring20.pngFortinet FortiGate.



Drive-by download rate 100.0% vbweb-verified.jpg
Malware block rate 100.0%
Phishing block rate 99.3%
False positive rate 2.6%


iBoss extends its impressive VBWeb performance by blocking all drive-by download cases (exploit kits) in this test, as well as all directly downloaded malware samples. iBoss also blocked over 99 per cent of phishing sites. The product easily earns its fifth VBWeb certification.



Kaspersky Web Traffic Security

Drive-by download rate 100.0% vbweb-verified.jpg
Malware block rate 99.0%
Phishing block rate 97.4%
False positive rate 0.0%


Kaspersky Web Traffic Security blocked all of the more than 650 exploit kits seen in this test. The detection rate of direct malware downloads was very good too, with only a handful of them missed. The product also achieved a good phishing blocking rate of over 97 per cent, thus showing that the gateway product provides an excellent first line of defence for web-based threats and its third VBWeb certification is fully deserved.

Kaspersky-VBWeb-Spring20.pngKaspersky Web Traffic Security.


Appendix: The Test Methodology

The test ran from 14 February 2020 to 29 February 2020, during which period we gathered a large number of URLs (most of which were found through public sources) which we had reason to believe could serve a malicious response. We opened the URLs in one of our test browsers, selected at random.

When our systems deemed the response sufficiently likely to fit one of various definitions of ‘malicious’, we made the same request in the same browser a number of times, each with one of the participating products in front of it. The traffic to the filters was replayed from our cache within seconds of the original request having been made, thus making it a fully real-time test.

We did not need to know at this point whether the response was actually malicious, thus our test didn’t depend on malicious sites that were already known to the security community. During a review of the test corpus some days later, we analysed the responses and discarded cases for which the traffic was not deemed malicious.

In this test, we checked products against 654 drive-by downloads (exploit kits), 482 direct malware downloads and 416 phishing sites, a category which also includes sites that trick the user into calling a phone number. To qualify for a VBWeb award, the weighted average catch rate of these two categories, with weights of 90% and 10% respectively, needed to be at least 80%.

The test focused on both HTTP and HTTPS traffic. It did not look at extremely targeted attacks or possible vulnerabilities in the products themselves.

Data from the test was provided by various public sources as well as an API provided by Active Defense6.


Test Machines

Each request was made from a randomly selected virtual machine using one of the available browsers. The machines ran either Windows XP Service Pack 3 Home Edition 2002, or Windows 7 Service Pack 1 Ultimate 2009 and all ran slightly out-of-date browsers and browser plug-ins.



1 See the regular VBSpam reports on the email-based threat landscape and email security products’ ability to protect email accounts: https://www.virusbulletin.com/testing/vbspam/.

2 https://nao-sec.org/2019/12/say-hello-to-bottle-exploit-kit.html

3 https://blog.trendmicro.com/trendlabs-security-intelligence/operation-overtrap-targets-japanese-online-banking-users-via-bottle-exploit-kit-and-brand-new-cinobi-banking-trojan/

4 https://www.virusbulletin.com/virusbulletin/2020/01/vbweb-comparative-review/

5 The requests are replayed in near real time.

6 https://www.activedefense.co.jp/adctd-api-spec/

Download PDF



Latest reviews:

VBSpam comparative review

The Q2 2024 VBSpam test measured the performance of ten full email security solutions, one custom configured solution and one open‑source solution.

VBSpam comparative review

The Q1 2024 VBSpam test measured the performance of nine full email security solutions, one custom configured solution and one open‑source solution.

VBSpam comparative review

The Q4 2023 VBSpam test measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q3 2023 VBSpam test we measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q2 2023 VBSpam test we measured the performance of nine full email security solutions, one custom configured solution, one open-source solution and one blocklist.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.