Impressive results in latest VBSpam test

Posted by   Virus Bulletin on   Feb 4, 2013

Excellent performances in spam filter test - plus evidence of a correlation between spam 'passing' SPF and an increased delivery rate.

No fewer than ten anti-spam solutions achieved a VBSpam+ award in VB's latest spam filter test, with another ten achieving a standard VBSpam award.

The results of the latest VBSpam test are good news for millions of users of spam filters. Spam filters have become pretty good at blocking spam and for this reason we felt it was appropriate to increase the threshold for obtaining a VBSpam award. Even having done this, 20 of the products in the test peformed well enough to achieve a VBSpam award, and of those, ten reached the required standard to earn a VBSpam+ award - showing a spam catch rate of 99.50% or more, without misclassifying any legitimate emails.

Of the products that achieved a VBSpam+ award, OnlyMyEmail, ZEROSPAM and Bitdefender had the highest spam catch rates, with OnlyMyEmail impressing the team by being the first product not to misclassify a single message in a corpus consisting of more than 70,000 emails.

SPF status

Alongside the product tests, the VBSpam team looked at the SPF status of the spam messages in the corpus to see if this had any impact on the likelihood of the messages being blocked.

SPF (Sender Policy Framework) is a mechanism designed to 'protect' the sending domain of the email, by indicating (via DNS) which IP addresses are authorized to send email from that domain. While SPF is no silver bullet for stopping spam, or even preventing the abuse of a sending domain, the use of SPF is recommended by organisations such as MAAWG, and most large senders have implemented an SPF policy.

Excluding domains for which the SPF record could not be parsed, or for which no DNS record was available when the email was sent, we distinguished five different SPF statuses: 'pass' ('this IP address is allowed to send email from this domain'); 'fail' ('this email is not from us'); 'softfail' ('it's probably not from us, but we can't be sure'); 'neutral' ('we don't know') and 'none' (no SPF record avialable). The table below shows how many products (out of 21) failed to block the emails with the various statuses.

SPF statusNo. of emailsNo. of products that failed to block emails (average)Standard deviation

As can be seen from the table, and possibly more clearly from the graph below, the is a correlation between the SPF status and the likelihood of a spam message being missed, with SPF fails more likely to be blocked and SPF passes more likely to be missed.


It is important to note that this doesn't necessarily imply a causal relationship: it may well be that the SPF passes were mostly emails sent from compromised accounts, which are harder to block for different reasons. Also, the results can be read to suggest that spammers should increase their usage of SPF if they want to increase their delivery rates. This may be true, but it may not necessarily be a bad thing: passing SPF merely adds to the veracity of the sending domain. If the domain belongs to a spammer, it will make blocking easier.

For Virus Bulletin subscribers, the full test report is available here. Non-subscribers can purchase the report as a standalone article ($19.95) here.

More on the VBSpam tests, including historical performance of the participating products, can be found here.

Posted on 04 February 2013 by Martijn Grooten



Latest posts:

In memoriam: Prof. Ross Anderson

We were very sorry to learn of the passing of Professor Ross Anderson a few days ago.

In memoriam: Dr Alan Solomon

We were very sorry to learn of the passing of industry pioneer Dr Alan Solomon earlier this week.

New paper: Nexus Android banking botnet – compromising C&C panels and dissecting mobile AppInjects

In a new paper, researchers Aditya K Sood and Rohit Bansal provide details of a security vulnerability in the Nexus Android botnet C&C panel that was exploited in order to gather threat intelligence, and present a model of mobile AppInjects.

New paper: Collector-stealer: a Russian origin credential and information extractor

In a new paper, F5 researchers Aditya K Sood and Rohit Chaturvedi present a 360 analysis of Collector-stealer, a Russian-origin credential and information extractor.

VB2021 localhost videos available on YouTube

VB has made all VB2021 localhost presentations available on the VB YouTube channel, so you can now watch - and share - any part of the conference freely and without registration.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.