A meta-analysis of recent malware tests

Richard Ford Florida Institute of Technology
Liam Mayron Florida Institute of Technology

Every year, some of the most heated discussions in the anti-malware world focus on the testing of anti-malware software. Vendors typically select tests they have done well in, and the smallest differences in detection rates are touted as absolute proof of superiority. However, while individual tests are often compared, there has been little work that attempts to analyse variance between tests and to determine from an objective and scientific perspective the importance of measured differences between product performance across tests.

In this paper, we conduct a meta-analysis of anti-malware tests for 2012 and 2013, using techniques which are commonly found in other disciplines, especially medicine. For example, statistical analysis methods, such as analysis of variance (ANOVA) can be applied to the data in order to determine if there is consistency between tests. Data mining and machine learning techniques will be applied to analyse and produce models of the data.

Using these techniques we can measure how consonant or dissonant each test is in comparison with 'average' test results. We also examine the statistical significance of differences in detection in an individual test. Is scoring slightly higher in one particular test a predictor of future performance in other tests, or are small differences simply a result of random variations? Furthermore, the manner in which test results are clustered may help identify outliers that provide different results from the majority of tests and merit further discussion. In essence, we show how meaningful rankings within anti-malware tests are, taken separately, and taken across multiple testers, as well as identify tests which seem to show differences that are not easily accounted for.

VB2013 takes place 2-4 October 2013 in Berlin, Germany.

The full programme for VB2013, including abstracts for each paper, can be viewed here.

Click here for more details about the conference.


We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.