Impressive results in latest VBSpam test

Posted by   Virus Bulletin on   Feb 4, 2013

Excellent performances in spam filter test - plus evidence of a correlation between spam 'passing' SPF and an increased delivery rate.

No fewer than ten anti-spam solutions achieved a VBSpam+ award in VB's latest spam filter test, with another ten achieving a standard VBSpam award.

The results of the latest VBSpam test are good news for millions of users of spam filters. Spam filters have become pretty good at blocking spam and for this reason we felt it was appropriate to increase the threshold for obtaining a VBSpam award. Even having done this, 20 of the products in the test peformed well enough to achieve a VBSpam award, and of those, ten reached the required standard to earn a VBSpam+ award - showing a spam catch rate of 99.50% or more, without misclassifying any legitimate emails.

Of the products that achieved a VBSpam+ award, OnlyMyEmail, ZEROSPAM and Bitdefender had the highest spam catch rates, with OnlyMyEmail impressing the team by being the first product not to misclassify a single message in a corpus consisting of more than 70,000 emails.

SPF status

Alongside the product tests, the VBSpam team looked at the SPF status of the spam messages in the corpus to see if this had any impact on the likelihood of the messages being blocked.

SPF (Sender Policy Framework) is a mechanism designed to 'protect' the sending domain of the email, by indicating (via DNS) which IP addresses are authorized to send email from that domain. While SPF is no silver bullet for stopping spam, or even preventing the abuse of a sending domain, the use of SPF is recommended by organisations such as MAAWG, and most large senders have implemented an SPF policy.

Excluding domains for which the SPF record could not be parsed, or for which no DNS record was available when the email was sent, we distinguished five different SPF statuses: 'pass' ('this IP address is allowed to send email from this domain'); 'fail' ('this email is not from us'); 'softfail' ('it's probably not from us, but we can't be sure'); 'neutral' ('we don't know') and 'none' (no SPF record avialable). The table below shows how many products (out of 21) failed to block the emails with the various statuses.

SPF statusNo. of emailsNo. of products that failed to block emails (average)Standard deviation
fail31710.240.04
pass81060.930.23
softfail86720.450.09
neutral134660.340.04
none269380.430.06

As can be seen from the table, and possibly more clearly from the graph below, the is a correlation between the SPF status and the likelihood of a spam message being missed, with SPF fails more likely to be blocked and SPF passes more likely to be missed.

VBSpam SPF

It is important to note that this doesn't necessarily imply a causal relationship: it may well be that the SPF passes were mostly emails sent from compromised accounts, which are harder to block for different reasons. Also, the results can be read to suggest that spammers should increase their usage of SPF if they want to increase their delivery rates. This may be true, but it may not necessarily be a bad thing: passing SPF merely adds to the veracity of the sending domain. If the domain belongs to a spammer, it will make blocking easier.

For Virus Bulletin subscribers, the full test report is available here. Non-subscribers can purchase the report as a standalone article ($19.95) here.

More on the VBSpam tests, including historical performance of the participating products, can be found here.

Posted on 04 February 2013 by Martijn Grooten

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest posts:

VB2021 localhost is over, but the content is still available to view!

VB2021 localhost - VB's second virtual conference - took place last week, but you can still watch all the presentations.

VB2021 localhost call for last-minute papers

The call for last-minute papers for VB2021 localhost is now open. Submit before 20 August to have your paper considered for one of the slots reserved for 'hot' research!

New article: Run your malicious VBA macros anywhere!

Kurt Natvig explains how he recompiled malicious VBA macro code to valid harmless Python 3.x code.

New article: Dissecting the design and vulnerabilities in AZORult C&C panels

In a new article, Aditya K Sood looks at the command-and-control (C&C) design of the AZORult malware, discussing his team's findings related to the C&C design and some security issues they identified.

VB2021 localhost call for papers: a great opportunity

VB2021 localhost presents an exciting opportunity to share your research with an even wider cross section of the IT security community around the world than usual, without having to take time out of your work schedule (or budget) to travel.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.