Continual feature selection: a cost effective method for enhancing the capabilities of enterprise spam solutions

Vipul Sharma, John Gardiner Myers, Steve Lewis Proofpoint

The effectiveness of content-based spam filters is directly related to the quality of the features used in the filter's classification model. Features are the specific attributes examined by the spam filter. Highly effective filters may employ an extremely large number of such features (on the order of hundreds of thousands), which can consume a significant amount of both storage space and classification time. In the ongoing battle between spammers and spam filter developers, new techniques and technologies are continually being introduced by both sides. This means that the number and importance of the features needed to classify spam accurately is subject to continual change. A given feature might be very important at one point in time, but become irrelevant after a few months as spam campaigns and their associated techniques change. Discarding on a regular basis features that have become ineffective ('bad features') will benefit the spam filter with reduced classification time (reduced model training time and email delivery time), reduced storage requirements, increased spam detection accuracy and a reduced risk of over-fitting of the model.

In this paper we discuss and benchmark several statistical methods for feature selection in spam filtering. We also discuss the properties of good and bad features in spam filtering. We report a significant improvement in both the filter's performance and effectiveness.


We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.