VBSpam Email Security Comparative Review June 2018

Martijn Grooten & Ionuţ Răileanu

Virus Bulletin

Copyright © 2018 Virus Bulletin


 

Introduction

In this test – which forms part of Virus Bulletin's continuously running security product test suite – 13 full email security solutions and eight blacklists of various kinds were assembled on the test bench to measure their performance against various streams of wanted, unwanted and malicious emails.

The news in these test reports tends to be good: email security products are an important first line of defence against the many email-borne threats and, especially against the bulk of opportunistic threats, they perform really well. The news in this report is no exception, with all 13 full solutions obtaining a VBSpam award and an impressive ten of them performing well enough to earn a VBSpam+ award.

 

Malicious attachments

Email continues to be a major delivery vector for malware. Though only a small portion of spam emails carry malware, it is these kinds of emails that people are most concerned about.

Amongst the emails with malicious attachments seen in our spam feeds, two trends were noticeable: first, there was the use of Excel Web Query (.iqy) files as email attachments. These are very short files, often barely more than a few lines long, which upon being clicked, open Excel and pull data into a new spreadsheet.

This data exploits Dynamic Data Exchange (DDE), a feature in Office regularly abused by malware, to execute some PowerShell code, which often downloads a final malicious payload.

vbspam-fig1-june18.png The content of an .iqy file seen in this month's test data. (The domain has since been taken down.)

The idea behind using a new kind of attachment in malicious spam campaigns is to bypass email security and anti-virus products that have not yet caught up with the trend – but this only gives attackers a very small window of opportunity, as security products usually catch up quickly. Indeed, while more than a third of the malware seen in this test was delivered as an .iqy attachment, few products had a hard time blocking these emails.

The same was true for the second trend we observed in malicious spam during the test period, though things here were more subtle: the sending of malicious Word attachments protected with a password. The password is sent as part of the body of the email, but an anti-virus scanner that is built into an email security product will only see the attachment and not its content1.

vbspam-fig2-june18.pngRecruitment spam has been a common delivery vector for password-protected Office attachments.

Many products would thus not have recognized the attachment as malware, though this test confirmed they would still have recognized the email as spam. Testing the user experience is currently beyond the remit of the VBSpam test, but we know that some implementations allow an end‑user to open emails that are merely marked as spam, while those that are detected as malware are put safely beyond the user's reach. In one infamous case2, a malicious email was put into quarantine by an email security product only to be retrieved from there by the end‑user – with far-reaching consequences.

In total, this test saw 1,146 emails with a malicious attachment. All but 60 of these were blocked by every email security product in the test.

One interesting point of note is a slight drop in the number of these emails that were picked up by IP-based blacklists. Such blacklists aren't exposed to attachments, so they can't be blamed for missing some of them, but the decrease in detections may also indicate that spammers are doing a better job at evading blacklisted IP addresses in these campaigns.

 

The spam that maybe isn't spam

The VBSpam set-up runs continuously throughout the year and many participating products – not just those that choose to be part of the quarterly public tests – use the weekly feedback we provide to improve their products.

One thing that regularly comes up in discussions with vendors is the issue of emails in the spam corpus that don't appear to be spam, and which the vendors claim some of their customers have actually subscribed to. Email security experts would counter that by saying that spam is about consent, not about content, and thus the same email can be both ham and spam to different customers.

Until now, emails in the spam feed that seemed legitimate and that were marked as legitimate by several products have been excluded from the test. That rule has now changed. Apart from transactional emails and NDRs, which will be excluded from the test corpus3, all emails sent as part of our spam feeds will be included in the test. However, those emails that appear legitimate in terms of both content and sender will be included in the spam corpus with a weight of 0.2.

We refer to these emails as 'unwanted', a name which is to be read as 'merely unwanted'. Obviously, all spam emails are unwanted, but in most cases they are far more undesirable.

 

Results

Yet again, the results of the test show that when it comes to volume, spam is an extremely well mitigated problem. Several products in this test blocked more than 99.9 per cent of the spam emails.

As mentioned in the introduction, all participating full solutions achieved a VBSpam award, with no fewer than ten performing well enough to achieve a VBSpam+ award. You will find all the details on the following pages, while a historic overview of products' performances can be found on our website: https://www.virusbulletin.com/testing/vbspam.

New in this test is Safemail, a cloud-based product from Italian company Spin. Though new to the test, the company has been working in the anti-abuse space since 1996. The product offers many features, most of which are beyond the scope of this test, but one worthy of mention is the option to set the product to 'paranoid' mode, where every DKIM and SPF fail is blocked.

What we did measure was the product's performance, and that was excellent: it achieved a spam catch rate of more than 99.9%, while avoiding false positives in both the ham and newsletter corpus – something achieved by only three other products in the test. It is thus well deserving of a VBSpam+ award on its debut appearance.

All products blocked at least 98 per cent of spam carrying malware, although performance here was slightly poorer than that on spam in general, suggesting that those using spam to spread malware make more of an effort to ensure their emails bypass spam filters. ESET, Libra Esva and OnlyMyEmail were the only products that blocked all spam carrying malware.

 

Axway MailGate 5.5.1

SC rate: 99.52%
FP rate: 0.00%
Final score: 99.47
Project Honey Pot SC rate: 99.33%
Abusix SC rate: 99.69%
Newsletters FP rate: 1.3%
Malware SC rate: 98.87%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

VBSpam+

 

Bitdefender Security for Mail Servers 3.1.6

SC rate: 99.96%
FP rate: 0.00%
Final score: 99.96
Project Honey Pot SC rate: 99.99%
Abusix SC rate: 99.94%
Newsletters FP rate: 0.0%
Malware SC rate: 98.08%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

ESET Mail Security for Microsoft Exchange Server

SC rate: 99.99%
FP rate: 0.00%
Final score: 99.99
Project Honey Pot SC rate: 99.998%
Abusix SC rate: 99.99%
Newsletters FP rate: 0.0%
Malware SC rate: 100.00%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

Forcepoint Email Security Cloud

SC rate: 99.14%
FP rate: 0.10%
Final score: 98.66
Project Honey Pot SC rate: 98.30%
Abusix SC rate: 99.85%
Newsletters FP rate: 0.0%
Malware SC rate: 99.04%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-verified-0618.jpg

 

Fortinet FortiMail

SC rate: 99.97%
FP rate: 0.00%
Final score: 99.97
Project Honey Pot SC rate: 99.998%
Abusix SC rate: 99.95%
Newsletters FP rate: 0.0%
Malware SC rate: 99.30%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

IBM Lotus Protector for Mail Security

SC rate: 99.96%
FP rate: 0.00%
Final score: 99.95
Project Honey Pot SC rate: 99.96%
Abusix SC rate: 99.96%
Newsletters FP rate: 0.3%
Malware SC rate: 99.56%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

Kaspersky for Exchange

SC rate: 99.98%
FP rate: 0.00%
Final score: 99.95
Project Honey Pot SC rate: 99.98%
Abusix SC rate: 99.98%
Newsletters FP rate: 0.7%
Malware SC rate: 99.83%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

Kaspersky Linux Mail Security 8.0

SC rate: 99.98%
FP rate: 0.00%
Final score: 99.95
Project Honey Pot SC rate: 99.98%
Abusix SC rate: 99.97%
Newsletters FP rate: 0.7%
Malware SC rate: 99.83%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

Libra Esva 4.1.0.0

SC rate: 99.96%
FP rate: 0.00%
Final score: 99.93
Project Honey Pot SC rate: 99.99%
Abusix SC rate: 99.94%
Malware SC rate: 100.00%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

OnlyMyEmail's Corporate MX-Defender

SC rate: 99.997%
FP rate: 0.01%
Final score: 99.79
Project Honey Pot SC rate: 99.996%
Abusix SC rate: 99.998%
Newsletters FP rate: 3.3%
Malware SC rate: 100.00%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-verified-0618.jpg

 

Spin Safemail 1.0

SC rate: 99.91%
FP rate: 0.00%
Final score: 99.91
Project Honey Pot SC rate: 99.90%
Abusix SC rate: 99.92%
Newsletters FP rate: 0.0%
Malware SC rate: 98.95%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

Trustwave

SC rate: 99.95%
FP rate: 0.00%
Final score: 99.87
Project Honey Pot SC rate: 99.93%
Abusix SC rate: 99.98%
Newsletters FP rate: 2.0%
Malware SC rate: 99.91%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-plus-0618.jpg

 

ZEROSPAM

SC rate: 99.91%
FP rate: 0.00%
Final score: 99.80
Project Honey Pot SC rate: 99.93%
Abusix SC rate: 99.89%
Newsletters FP rate: 2.7%
Malware SC rate: 99.21%
Speed:

10% GREEN 50% GREEN 95% GREEN 98% GREEN

vbspam-verified-0618.jpg

 

IBM X-Force Combined

SC rate: 96.89%
FP rate: 0.00%
Final score: 96.89
Project Honey Pot SC rate: 94.69%
Abusix SC rate: 98.76%
Newsletters FP rate: 0.0%
Malware SC rate: 75.22%

 

IBM X-Force IP

SC rate: 94.30%
FP rate: 0.00%
Final score: 94.30
Project Honey Pot SC rate: 89.82%
Abusix SC rate: 98.11%
Newsletters FP rate: 0.0%
Malware SC rate: 75.13%

 

IBM X-Force URL

SC rate: 61.71%
FP rate: 0.00%
Final score: 61.71
Project Honey Pot SC rate: 75.45%
Abusix SC rate: 50.04%
Newsletters FP rate: 0.0%
Malware SC rate: 0.96%

 

Spamhaus DBL

SC rate: 45.84%
FP rate: 0.01%
Final score: 45.77
Project Honey Pot SC rate: 47.92%
Abusix SC rate: 44.07%
Newsletters FP rate: 0.0%
Malware SC rate: 29.58%

 

Spamhaus ZEN

SC rate: 93.83%
FP rate: 0.00%
Final score: 93.83
Project Honey Pot SC rate: 88.74%
Abusix SC rate: 98.15%
Newsletters FP rate: 0.0%
Malware SC rate: 80.98%

 

Spamhaus ZEN+DBL

SC rate: 96.03%
FP rate: 0.01%
Final score: 95.96
Project Honey Pot SC rate: 93.29%
Abusix SC rate: 98.35%
Newsletters FP rate: 0.0%
Malware SC rate: 82.46%

 

URIBL (MX Tools)

SC rate: 47.76%
FP rate: 0.00%
Final score: 47.75
Project Honey Pot SC rate: 47.19%
Abusix SC rate: 48.24%
Newsletters FP rate: 0.3%
Malware SC rate: 15.18%

 

Zetascan (MX Tools)

SC rate: 93.37%
FP rate: 0.25%
Final score: 91.95
Project Honey Pot SC rate: 86.08%
Abusix SC rate: 99.56%
Newsletters FP rate: 4.3%
Malware SC rate: 90.92%

 

Results tables

  True negatives False positives FP rate False negatives True positives SC rate VBSpam Final score
Axway 7232 0 0.00% 577.4 120573.8 99.52% VBSpam+ 99.47
Bitdefender 7232 0 0.00% 46.6 121104.6 99.96% VBSpam+ 99.96
ESET 7232 0 0.00% 8.4 121142.8 99.99% VBSpam+ 99.99
Forcepoint 7225 7 0.10% 1047.2 120104 99.14% VBSpam 98.66
FortiMail 7232 0 0.00% 34.6 121116.6 99.97% VBSpam+ 99.97
IBM 7232 0 0.00% 45.4 121105.8 99.96% VBSpam+ 99.95
Kaspersky for Exchange 7232 0 0.00% 26.4 121124.8 99.98% VBSpam+ 99.95
Kaspersky LMS 7232 0 0.00% 28.8 121122.4 99.98% VBSpam+ 99.95
Libra Esva 7232 0 0.00% 47 121104.2 99.96% VBSpam+ 99.93
OnlyMyEmail 7231 1 0.01% 3.2 121148 99.997% VBSpam 99.79
Safemail 7232 0 0.00% 107.6 121043.6 99.91% VBSpam+ 99.91
Trustwave 7232 0 0.00% 54.6 121096.6 99.95% VBSpam+ 99.87
ZEROSPAM 7232 0 0.00% 110.4 121040.8 99.91% VBSpam 99.80
IBM X-Force Combined* 7232 0 0.00% 3767.8 117383.4 96.89% N/A 96.89
IBM X-Force IP* 7232 0 0.00% 6906.2 114245 94.30% N/A 94.30
IBM X-Force URL* 7232 0 0.00% 46382.8 74768.4 61.71% N/A 61.71
Spamhaus DBL* 7231 1 0.01% 65613 55538.2 45.84% N/A 45.77
Spamhaus ZEN* 7232 0 0.00% 7478.2 113673 93.83% N/A 93.83
Spamhaus ZEN+DBL* 7231 1 0.01% 4814 116337.2 96.03% N/A 95.96
URIBL* 7232 0 0.00% 63289.2 57862 47.76% N/A 47.75
Zetascan* 7214 18 0.25% 8035.8 113115.4 93.37% N/A 91.95

*The IBM X-Force, Spamhaus, URIBL and Zetascan products are partial solutions and their performance should not be compared with that of other products.
(Please refer to the text for full product names and details.)

 

  Newsletters Malware Project Honey Pot Abusix STDev Speed
False positives FP rate False negatives SC rate False negatives SC rate False negatives SC rate 10% 50% 95% 98%
Axway 4 1.3% 13 98.87%  371.8 99.33%  205.6 99.69%  1.05 Green Green Green Green
Bitdefender 0 0.0% 22 98.08%  5.2 99.99%  41.4 99.94%  0.33 Green Green Green Green
ESET 0 0.0% 0 100.00% 1.2 99.998%  7.2 99.99%  0.25 Green Green Green Green
Forcepoint 0 0.0% 11 99.04%  948.6 98.30%  98.6 98.30%  1.47 Green Green Green Green
FortiMail 0 0.0% 8 99.30%  1.2 99.998%  33.4 99.95%  0.47 Green Green Green Green
IBM 1 0.3% 5 99.56%  21.6 99.96%  23.8 99.96%  0.3 Green Green Green Green
Kaspersky for Exchange 2 0.7% 2  99.83% 10.6 99.98%  15.8 99.98%  0.3 Green Green Green Green
Kaspersky LMS 2 0.7% 2 99.83%  9 99.98% 19.8  99.97%  0.26 Green Green Green Green
Libra Esva 2 0.7% 0 100.00% 6.6 99.99%  40.4 99.94%  0.35 Green Green Green Green
OnlyMyEmail 10 3.3% 0 100.00% 2 99.996%  1.2 99.998%  0.07 Green Green Green Green
Safemail 0 0.0% 12 98.95%  54.6 99.90%  53 99.92%  0.53 Green Green Green Green
Trustwave 6 2.0% 1 99.91%  39.4 99.93%   15.2 99.98%  0.33 Green Green Green Green
ZEROSPAM 8 2.7% 9 99.21%  37.2  99.93%  73.2 99.89%  0.46 Green Green Green Green
IBM X-Force Combined* 0 0.0% 284 75.22%  2955.6  94.69% 812.2  98.76% 3.78  N/A N/A N/A N/A
IBM X-Force IP* 0 0.0% 285  75.13% 5668.6  89.82% 1237.6  98.11% 4.88  N/A N/A N/A N/A
IBM X-Force URL* 0 0.0% 1135 0.96% 13665.8 75.45% 32717 50.04% 21.84 N/A N/A N/A N/A
Spamhaus DBL* 0 0.0% 807 29.58%  28985.6  47.92% 36627.4  44.07% 20.95  N/A N/A N/A N/A
Spamhaus ZEN* 0 0.0% 218 80.98%  6264.8  88.74% 1213.4  98.15% 5.63  N/A N/A N/A N/A
Spamhaus ZEN+DBL* 0 0.0% 201 82.46%  3733.6  93.29%  1080.4 98.35%  4.15 N/A N/A N/A N/A
URIBL* 1 0.3% 972  15.18%  29392.8  47.19% 33896.4  48.24% 20.84  N/A N/A N/A N/A
Zetascan* 13 4.3% 104 90.92%  7748 86.08%  287.8 99.56%  7.16 N/A N/A N/A N/A

*The Spamhaus products, IBM X-Force and URIBL are partial solutions and their performance should not be compared with that of other products. None of the queries to the IP blacklists included any information on the attachments; hence their performance on the malware corpus is added purely for information.
The standard deviation of a product is calculated using the set of its hourly spam catch rates.

speed-colour-blobs-GREEN.jpg 0-30 seconds speed-colour-blobs-YELLOW.jpg 30 seconds to two minutes speed-colour-blobs-ORANGE.jpg two minutes to 10 minutes speed-colour-blobs-RED.jpg more than 10 minutes


(Please refer to the text for full product names.)

 

Hosted solutions Anti-malware IPv6 DKIM SPF DMARC Multiple MX-records Multiple locations
Forcepoint Forcepoint Advanced Malware Detection  
OnlyMyEmail Proprietary (optional)   *
Safemail ClamAV; proprietary
ZEROSPAM ClamAV      

* OnlyMyEmail verifies DMARC status but doesn't provide feedback at the moment.
(Please refer to the text for full product names.)

 

Local solutions Anti-malware IPv6 DKIM SPF DMARC Interface
CLI GUI Web GUI API
Axway MailGate Kaspersky, McAfee        
Bitdefender Bitdefender        
ESET ESET Threatsense    
FortiMail Fortinet  
IBM Sophos; IBM Remote Malware Detection          
Kaspersky for Exchange Kaspersky Lab        
Kaspersky LMS Kaspersky Lab      
Libra Esva ClamAV; others optional        
Trustwave Support for multiple third-party engines  √ √ 

(Please refer to the text for full product names.)

 

VBSpam-quadrant-June18.jpg

 (Please refer to the text for full product names.)

 

 

Appendix: set-up, methodology and email corpora

The full VBSpam test methodology can be found at https://www.virusbulletin.com/testing/vbspam/vbspam-methodology/.

The test ran for 19 days, from 12am on 12 May to 12am on 31 May 2018. A power cut caused a disruption between 18 and 20 May; hence emails sent during this period were excluded from the test.

The test corpus consisted of 128,775 emails. 121,244 of these were spam, 55,714 of which were provided by Project Honey Pot, with the remaining 65,530 spam emails provided by Abusix. There were 7,232 legitimate emails ('ham') and 299 newsletters.

116 emails in the spam corpus were considered 'unwanted' (emails contained in the spam feed that appeared legitimate in terms of both content and sender) and were included with a weight of 0.2; this explains the non-integer numbers in some of the tables.

Moreover, 1,146 emails from the spam corpus were found to contain a malicious attachment; though we report separate performance metrics on this corpus, it should be noted that these emails were also counted as part of the spam corpus.

Emails were sent to the products in real time and in parallel. Though products received the email from a fixed IP address, all products had been set up to read the original sender's IP address as well as the EHLO/HELO domain sent during the SMTP transaction, either from the email headers or through an optional XCLIENT SMTP command4. Consequently, products were able to filter email in an environment that was very close to one in which they would be deployed in the real world.

For those products running in our lab, we ran them as virtual machines on a VMware ESXi cluster. As different products have different hardware requirements – not to mention those running on their own hardware, or those running in the cloud – there is little point comparing the memory, processing power or hardware the products were provided with; we followed the developers' requirements and note that the amount of email we receive is representative of that received by a small organization.

Although we stress that different customers have different needs and priorities, and thus different preferences when it comes to the ideal ratio of false positives to false negatives, we created a one-dimensional 'final score' to compare products. This is defined as the spam catch (SC) rate minus five times the weighted false positive (WFP) rate. The WFP rate is defined as the false positive rate of the ham and newsletter corpora taken together, with emails from the latter corpus having a weight of 0.2:

WFP rate = (#false positives + 0.2 * min(#newsletter false positives , 0.2 * #newsletters)) / (#ham + 0.2 * #newsletters)

While in the spam catch rate (SC), emails considered 'unwanted' (see above) are included with a weight of 0.2.

The final score is then defined as:

Final score = SC - (5 x WFP)

In addition, for each product, we measure how long it takes to deliver emails from the ham corpus (excluding false positives) and, after ordering these emails by this time, we colour-code the emails at the 10th, 50th, 95th and 98th percentiles:

speed-colour-blobs-GREEN.jpg (green) = up to 30 seconds
YELLOW (yellow) = 30 seconds to two minutes
speed-colour-blobs-ORANGE.jpg (orange) = two to ten minutes
speed-colour-blobs-RED.jpg (red) = more than ten minutes

 

Products earn VBSpam certification if the value of the final score is at least 98 and the 'delivery speed colours' at 10 and 50 per cent are green or yellow and that at 95 per cent is green, yellow or orange.

Meanwhile, products that combine a spam catch rate of 99.5% or higher with a lack of false positives, no more than 2.5% false positives among the newsletters and 'delivery speed colours' of green at 10 and 50 per cent and green or yellow at 95 and 98 per cent earn a VBSpam+ award.

 

Footnotes

1 It is technically possible for a product to read the password from the email body and use it to decrypt the attachment, but this is generally considered bad practice.

2 https://www.f-secure.com/weblog/archives/00002226.html.

3 We do not believe it is reasonable to expect these emails to be filtered correctly in the VBSpam test environment.

4 http://www.postfix.org/XCLIENT_README.html.

Download PDF

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest reviews:

VBSpam comparative review

The Q1 2024 VBSpam test measured the performance of nine full email security solutions, one custom configured solution and one open‑source solution.

VBSpam comparative review

The Q4 2023 VBSpam test measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q3 2023 VBSpam test we measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q2 2023 VBSpam test we measured the performance of nine full email security solutions, one custom configured solution, one open-source solution and one blocklist.

VBSpam comparative review

In the Q1 2023 VBSpam test we measured the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.