Why there’s no one test to rule them all


Lysa Myers

West Coast Labs, USA
Editor: Helen Martin


‘Because every product has strengths and weaknesses, having a variety of different tests is essential.’ Lysa Myers, West Coast Labs

Anti-malware products are all alike the world over – with the same tactics, usage, features, speed of updates and target market, right? If that were true it would stand to reason that there would be only one or two types of appropriate tests to put those products through their paces. Just running a large number of threats and clean items against the different companies’ products would be sufficient. In reality, though, that is not the case.

It’s my position that there is no ‘One Test to Rule Them All’. The overarching objective of all tests is to emulate what users do in the real world. But users in China will have a different set-up from those in Germany, just as users in major banks will differ from home users with mobile anti-malware products. The threats that affect them differ, as does the information they want.

Similarly, the consumers of tests have interests in different types of products as well as different information. Anti-malware vendors themselves are consumers of tests. Their interests are similar in many ways to those of a user, but not identical. (After all, there is no financial incentive for users, regardless of a test’s outcome.)

So what should testers be doing? First, I believe there is still value in what are now considered ‘traditional’ testing methods. Especially with new and emerging markets (both geographically and technologically), periodic static testing can function as a baseline to indicate which solutions are valid anti-malware products. There may come a time when anti-malware scanner technology has changed so much that this is no longer adequate, but until then static tests remain a good way to validate basic functionality.

Beyond that, things get more complex. While there is a lot of the traditional technology in modern anti-malware products, there are also a lot of new modules and features. While most folks agree to a certain extent on what an anti-malware product looks like, not everyone agrees what constitutes newer technologies. Testers must often make decisions regarding what qualifies as a Standard Newfangled Widget when different vendors come up with different ways of going about things. Anti-spyware and anti-spam are excellent examples of how this has played out in the past. Testers had to make decisions, with a significant amount of input from vendors, as to what samples were appropriate and how they needed to be addressed. Technologies like IPS/IDS or DLP make this more complicated still, as they bear less resemblance to signature scanners.

Because of the speed and prevalence of malware, time is one of the most essential elements. Scans on users’ machines don’t happen only quarterly or monthly, so the frequency of tests has increased. As the testing time decreases, the relevance of samples becomes vastly more important.

People don’t only use on-access or on-demand scanners, but also run-time detection such as behavioural scanners and emulators. Most people in the anti-malware industry these days agree that dynamic testing is essential.

Different testers may also choose to validate detection in various other ways as well. For example, retrospective testing examines scanners’ abilities beyond simply detecting malware which is already known. Those products with exceptional heuristic or ‘generic’ detection capabilities can differentiate themselves here.

There are also concerns which go beyond the accuracy of detection, but which are nevertheless important to users. Performance testing in the sense of memory/CPU usage can reassure users that, during scanning, their machine will not be disproportionately affected – they can see that they don’t need to sacrifice usability for thoroughness of protection.

Because every product has strengths and weaknesses, having a variety of different tests is essential. You must have a wide and varied vocabulary to describe things to people in a way that is meaningful to the majority. Let us not limit our vocabularies to just a few adjectives, but strive to serve and create an erudite user base.



Latest articles:

Cryptojacking on the fly: TeamTNT using NVIDIA drivers to mine cryptocurrency

TeamTNT is known for attacking insecure and vulnerable Kubernetes deployments in order to infiltrate organizations’ dedicated environments and transform them into attack launchpads. In this article Aditya Sood presents a new module introduced by…

Collector-stealer: a Russian origin credential and information extractor

Collector-stealer, a piece of malware of Russian origin, is heavily used on the Internet to exfiltrate sensitive data from end-user systems and store it in its C&C panels. In this article, researchers Aditya K Sood and Rohit Chaturvedi present a 360…

Fighting Fire with Fire

In 1989, Joe Wells encountered his first virus: Jerusalem. He disassembled the virus, and from that moment onward, was intrigued by the properties of these small pieces of self-replicating code. Joe Wells was an expert on computer viruses, was partly…

Run your malicious VBA macros anywhere!

Kurt Natvig wanted to understand whether it’s possible to recompile VBA macros to another language, which could then easily be ‘run’ on any gateway, thus revealing a sample’s true nature in a safe manner. In this article he explains how he recompiled…

Dissecting the design and vulnerabilities in AZORult C&C panels

Aditya K Sood looks at the command-and-control (C&C) design of the AZORult malware, discussing his team's findings related to the C&C design and some security issues they identified during the research.

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.