VBWeb Comparative Review - Autumn 2018

Martijn Grooten & Adrian Luca

Virus Bulletin

Copyright © Virus Bulletin 2018



Together with email [1], the web is one of the two major infection vectors through which organizations and individuals get infected with malware, and often the two go hand in hand, with email-based threats linking to web‑based malware. However, some attacks exist on the web only, and recently there has been an uptick in exploit kit activity, if not in the number of attacks then at least in the number of active kits: Malwarebytes' most recent report [2] lists five kits that were active at some point during the autumn.

To protect against such exploit kits, regular patching remains the best strategy and it is encouraging to note that automatic patching is becoming increasingly common. But as a backstop against forgotten or ignored patches, and as protection against web-based malware that uses social engineering, web security products are an important protection layer.


The Autumn 2018 threat landscape

Though the prevalence of exploit kits is nowhere near its peak of a few years ago, the threat landscape remains varied, with more than half a dozen different kits seen during this test – suggesting there is still money to be made by cybercriminals this way.

All the known active kits were seen during this test with the exception of the very geographically targeted Magnitude. One particularly noteworthy kit was Fallout, a new exploit kit, first seen in August 2018 [3], that is based on the code of the old Nuclear exploit kit. Interestingly, we saw some instances of Fallout being missed in this test.

A large range of malware was observed in this test, including the new Kraken Cryptor ransomware [4].

kraken_cryptor_oct2018.pngKraken Cryptor.

Thanks to the availability of free TLS certificates, we continue to see many threats using HTTPS to deliver their payload. Though the privacy implications of an organization intercepting HTTPS for its employees should be well understood, it is important to note that not doing so would mean that a not insignificant amount of malicious web traffic would bypass the security product in place.

The recent threat of web-based illicit cryptocurrency miners [5] continues to be seen; we spotted more than 450 instances of such threats during the test. However, despite the various obfuscation applied by these threats, we found that the products we tested blocked almost all of them.



Fortinet FortiGate

Drive-by download rate
85.6% vbweb-verified.jpg
Malware block rate 99.3%
Weighted average 87.0% 
Potentially malicious rate 96.5% 
Cryptocurrency miner block rate 100.0% 
False positive rate

As in previous tests, Fortinet's FortiGate appliance performed well in this test, easily achieving its seventh VBWeb award and justifying its customers' trust in the product.

However, FortiGate did have some issues with some of the earlier instances of the new Fallout exploit kit seen during the test, which reduced its drive-by download catch rate. There were fewer issues with later instances of the same threat though, and the product did block the increasingly common threat of malware downloads very effectively.

fortiguard_21_sept_2018_2.pngFortinet FortiGate.


Kaspersky Web Traffic Security 6.0

Drive-by download rate
100.0%  vbweb-verified.jpg
Malware block rate 99.2% 
Weighted average 99.9%
Potentially malicious rate 99.2%
Cryptocurrency miner block rate 100.0%
False positive rate

Kaspersky Lab is hardly a new name in IT security, yet the company's submission of Kaspersky Web Traffic Security to this test marks its first public participation in our VBWeb tests. The 6.0 version of the product was released in October 2018.

Its performance was impressive – it was the only product to block all exploit kits, and also blocked almost all direct downloads of malware. The product's first VBWeb award is thus fully deserved and Kaspersky's customers can feel confident in the product's ability to protect them from web‑based threats.

kaspersky_19_oct_2018_1.pngKaspersky Web Traffic Security.


Trustwave Secure Web Gateway

Drive-by download rate
95.3%  vbweb-verified.jpg
Malware block rate 98.6% 
Weighted average 95.7% 
Potentially malicious rate 94.2% 
Cryptocurrency miner block rate 100.0% 
False positive rate

Trustwave's Secure Web Gateway product is a regular in the VBWeb tests, always performing well. This test was no exception, with high block rates in all categories.

There were some misses among earlier instances of the new Fallout exploit kit, but after that all instances of Fallout were consistently blocked. Yet another VBWeb award, the product's sixth, was easily achieved and the product's good reputation once again affirmed.

trustwave_19_oct_2018_1.pngTrustwave Secure Web Gateway.


Appendix: The test methodology

The test ran from 10 September to 23 October 2018 (with a short break in the early part of October when the team attended the VB Conference), during which period we gathered a large number of URLs (most of which were found through public sources) which we had reason to believe could serve a malicious response. We opened the URLs in one of our test browsers, selected at random.

When our systems deemed the response sufficiently likely to fit one of various definitions of 'malicious', we made the same request in the same browser a number of times, each with one of the participating products in front of it. The traffic to the filters was replayed from our cache within seconds of the original request having been made, thus making it a fully real-time test.

We did not need to know at this point whether the response was actually malicious, thus our test didn't depend on malicious sites that were already known to the security community. During a review of the test corpus some days later, we analysed the responses and discarded cases for which the traffic was not deemed malicious.

In this test, we checked products against 257 drive‑by downloads (exploit kits) and 1,452 direct malware downloads. To qualify for a VBWeb award, the weighted average catch rate of these two categories, with weights of 90% and 10% respectively, needed to be at least 70%.

We also checked the products against 1,013 URLs that we call 'potentially malicious'. These were URLs for which we had strong evidence that they would serve a malicious response in some cases, but they didn't when we requested it. There could be a number of reasons for this, from server-side randomness to our test lab being detected by anti-analysis tools.

While one can have a perfectly good web security product that doesn't block any of these, we believe that blocking such URLs can serve as an indication of a product's ability to block threats proactively without inspecting the traffic. For some customers this could be important, and for developers this is certainly valuable information, hence we decided to include it in this and future reports.

The test focused on unencrypted HTTP traffic. It did not look at extremely targeted attacks or possible vulnerabilities in the products themselves.


Test machines

Each request was made from a randomly selected virtual machine using one of the available browsers. The machines ran either Windows XP Service Pack 3 Home Edition 2002 or Windows 7 Service Pack 1 Ultimate 2009, and all ran slightly out-of-date browsers and browser plug-ins.



[1]  See the regular VBSpam reports on the email-based threat landscape and email security products’ ability to protect email accounts:

[2] https://blog.malwarebytes.com/threat-analysis/2018/10/exploit-kits-fall-2018-review/.

[3] https://www.nao-sec.org/2018/09/hello-fallout-exploit-kit.html.

[4] https://www.recordedfuture.com/kraken-cryptor-ransomware/.

[5] https://www.virusbulletin.com/blog/2018/10/vb2018-paper-drive-download-drive-mining-understanding-new-paradigm/.

Download PDF



Latest reviews:

VBSpam comparative review

The Q1 2024 VBSpam test measured the performance of nine full email security solutions, one custom configured solution and one open‑source solution.

VBSpam comparative review

The Q4 2023 VBSpam test measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q3 2023 VBSpam test we measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q2 2023 VBSpam test we measured the performance of nine full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q1 2023 VBSpam test we measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.