'Hotmail and Gmail have best spam filter' says Cascade spam test

Posted by   Virus Bulletin on   Feb 15, 2012

Comparative test did not take false positives into account.

Researchers from Cascade Insights performed a comparative spam filtering test on the three major webmail providers and concluded that Hotmail performed best, shortly followed by Gmail, with Yahoo! a distant third.

The researchers registered accounts at all three providers and, for comparison, a fourth one at an ISP that did not perform any kind of filtering. They then seeded these accounts by sending 'replies' to known spammers, posting the addresses on public sites and submitting them to dubious-looking websites.

For the test period, the researchers manually classified all the emails they received, marking as 'ham' all legitimate emails as well as those that were solicited - even if they were scams. They then calculated the percentage of spam in each of the inboxes and found that a little less than half of the email in Hotmail's inbox was spam, a ratio almost matched by Gmail. In Yahoo!'s case, over 58 per cent of the inbox consisted of spam, compared to close to two thirds for the unfiltered account.

The researchers did not look at false positive rates. While they were right to say that the emails they classified as 'ham' (many of which were of dubious nature) weren't representative of a real stream of legitimate emails, it is a shame that they didn't find another way to send legitimate mail to the accounts - avoiding false positives is an important part of running a spam filter and a good spam catch rate is useless if many legitimate emails are blocked as well.

A possibly more serious flaw in the test was the way performance was measured. Measuring the percentage of spam in the inbox is a nice idea. After all, users of free webmail services care little about the amount of spam sent to them; rather, they care about the amount of spam that makes it into their inbox. However, in using this method, the researchers implicitly take the 'ham' in their mail streams into account - the same messages that they previously claimed could not be used for measuring false positives.

The full report, which includes a detailed description of the methodology used, can be download as a PDF here.

Posted on 15 February 2012 by Virus Bulletin



Latest posts:

New paper: Collector-stealer: a Russian origin credential and information extractor

In a new paper, F5 researchers Aditya K Sood and Rohit Chaturvedi present a 360 analysis of Collector-stealer, a Russian-origin credential and information extractor.

VB2021 localhost videos available on YouTube

VB has made all VB2021 localhost presentations available on the VB YouTube channel, so you can now watch - and share - any part of the conference freely and without registration.

VB2021 localhost is over, but the content is still available to view!

VB2021 localhost - VB's second virtual conference - took place last week, but you can still watch all the presentations.

VB2021 localhost call for last-minute papers

The call for last-minute papers for VB2021 localhost is now open. Submit before 20 August to have your paper considered for one of the slots reserved for 'hot' research!

New article: Run your malicious VBA macros anywhere!

Kurt Natvig explains how he recompiled malicious VBA macro code to valid harmless Python 3.x code.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.