Why there’s no one test to rule them all


Lysa Myers

West Coast Labs, USA
Editor: Helen Martin


‘Because every product has strengths and weaknesses, having a variety of different tests is essential.’ Lysa Myers, West Coast Labs

Anti-malware products are all alike the world over – with the same tactics, usage, features, speed of updates and target market, right? If that were true it would stand to reason that there would be only one or two types of appropriate tests to put those products through their paces. Just running a large number of threats and clean items against the different companies’ products would be sufficient. In reality, though, that is not the case.

It’s my position that there is no ‘One Test to Rule Them All’. The overarching objective of all tests is to emulate what users do in the real world. But users in China will have a different set-up from those in Germany, just as users in major banks will differ from home users with mobile anti-malware products. The threats that affect them differ, as does the information they want.

Similarly, the consumers of tests have interests in different types of products as well as different information. Anti-malware vendors themselves are consumers of tests. Their interests are similar in many ways to those of a user, but not identical. (After all, there is no financial incentive for users, regardless of a test’s outcome.)

So what should testers be doing? First, I believe there is still value in what are now considered ‘traditional’ testing methods. Especially with new and emerging markets (both geographically and technologically), periodic static testing can function as a baseline to indicate which solutions are valid anti-malware products. There may come a time when anti-malware scanner technology has changed so much that this is no longer adequate, but until then static tests remain a good way to validate basic functionality.

Beyond that, things get more complex. While there is a lot of the traditional technology in modern anti-malware products, there are also a lot of new modules and features. While most folks agree to a certain extent on what an anti-malware product looks like, not everyone agrees what constitutes newer technologies. Testers must often make decisions regarding what qualifies as a Standard Newfangled Widget when different vendors come up with different ways of going about things. Anti-spyware and anti-spam are excellent examples of how this has played out in the past. Testers had to make decisions, with a significant amount of input from vendors, as to what samples were appropriate and how they needed to be addressed. Technologies like IPS/IDS or DLP make this more complicated still, as they bear less resemblance to signature scanners.

Because of the speed and prevalence of malware, time is one of the most essential elements. Scans on users’ machines don’t happen only quarterly or monthly, so the frequency of tests has increased. As the testing time decreases, the relevance of samples becomes vastly more important.

People don’t only use on-access or on-demand scanners, but also run-time detection such as behavioural scanners and emulators. Most people in the anti-malware industry these days agree that dynamic testing is essential.

Different testers may also choose to validate detection in various other ways as well. For example, retrospective testing examines scanners’ abilities beyond simply detecting malware which is already known. Those products with exceptional heuristic or ‘generic’ detection capabilities can differentiate themselves here.

There are also concerns which go beyond the accuracy of detection, but which are nevertheless important to users. Performance testing in the sense of memory/CPU usage can reassure users that, during scanning, their machine will not be disproportionately affected – they can see that they don’t need to sacrifice usability for thoroughness of protection.

Because every product has strengths and weaknesses, having a variety of different tests is essential. You must have a wide and varied vocabulary to describe things to people in a way that is meaningful to the majority. Let us not limit our vocabularies to just a few adjectives, but strive to serve and create an erudite user base.



Latest articles:

VB2018 paper: Internet balkanization: why are we raising borders online?

Nowadays, walls are not just being raised in the real world, but on the Internet as well. Countries want to isolate themselves and shut down the information they are not comfortable with, or the companies they don’t want to do business with. Freedom…

VB2018 paper: Where have all the good hires gone?

Much ink has been spilled on the subject of the information security skills gap, and how difficult it is to hire and retain people for these positions. And yet, we all know someone who has had a hard time finding a suitable position despite having…

VB2018 paper: Little Brother is watching – we know all your secrets!

In their research, Siegfried Rasthofer, Stephan Huber & Steven Arzt evaluated the security level of the most popular family-tracking apps on Android. They assessed the security of the respective apps and conducted assessments of the corresponding…

VB2018 paper: Inside Formbook infostealer

Formbook is an infostealer that has been advertised for sale in public hacking forums since February 2016 by a user with the handle ‘ng-Coder' but only came to public attention after it was extensively used in spam campaigns in late 2017. This paper…

VB2018 paper: From Hacking Team to hacked team to...?

In this paper (presented at VB2018), Filip Kafka looks at the resurfaced Hacking Team spyware, and at what has changed since the company behind it faced a number of prominent hacks.

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.