Throwback Thursday: The Unbearable Lightness of Testing (December 1996)


Ian Whalley

Virus Bulletin, UK
Editor: Martijn Grooten


Back in 1996, the memory limits of the DOS environment posed issues for anti-malware developers that we wouldn't give a second thought to today. While scanners were already "groaning" under the load of the ever-increasing number of viruses (the growth in the number of known viruses was then around 150-200 per month), the need to add complex new scan capabilities - for dealing with macro viruses - threatened to be the last straw for some. The solution for several products of the time was to supply a second executable offering macro-scanning functionality. Then Editor of VB Ian Whalley quite rightly argued that it was unreasonable to expect people to have to run multiple programs to detect different types of virus and wondered where we would be if we had to have a product consisting of 9,500 separate executables, "one for every virus..."

(This article was first published in Virus Bulletin in December 1996.)

January 1997 sees the publication of the biannual VB DOS Scanner Comparative. Despite the rise of Windows 95 and NT, this is still very much our flagship review; the one which garners most attention. It has figures quoted from it left, right and centre, and, inevitably, attracts the most criticism.

The testing of anti-virus products is, as has been well documented in these and other pages in previous years, an incredibly difficult thing to do – or at least, to do well. VB is fortunate in being one of the few well-regarded organisations to perform comparative testing: it is not least for this reason that I take such care when carrying out these reviews.

As I write, I have reached the stage where preliminary sets of results are sent to developers in each company – this system has proven worthwhile in the past as far as catching small errors at an early stage goes, and offers time for the results to be discussed and suggestions to be made. Having valuable ideas put forward after publication is not immediately useful, after all…

Say what you like about the people who make virus scanners, each company certainly cares greatly how its product is reviewed. This initial, very restricted, distribution of results inevitably leads to a manifold increase in the level of email coming into my computers, and a flurry of extra checking of virus samples inevitably ensues as developers compare the results of their internal tests with ours.

During this period, there can be a certain antagonism between myself and the developers; testing methodology is often a sticking point. This time, the discussion has focused on the treatment of macro viruses. When it comes to adding new features to a scanner (e.g. the ability to scan with OLE2 Word documents), the DOS environment offers more problems than most, the most significant of which is the 640K memory limit.

With scanners already groaning under the load of the ever-increasing number of viruses, adding complex new scan capabilities threatens to be the last straw for some. The clear stopgap solution is to provide a second executable. Simply place the macro-scanning functionality in a program on its own, and the problem is solved – indeed, several products in this January’s tests do this. However, this system has drawbacks. First, re-educating a product’s users to execute two programs instead of the one they had to run previously will take time – it is inconvenient for these users to have to adjust their behaviour in this way. And is a one-stop solution too much to ask?

My problem with this particular solution is on a simpler level. Should the detection rate of the product’s macro add-in be included in the product’s score? That is to say, should the macro detector be run over the macro samples and that score added to that of the main program on the more traditional parasitic and boot sector viruses? My conclusion at this stage is that it should not. It is unreasonable to expect people to have to run multiple programs to detect different types of virus – it smacks of the thin end of the wedge. To take the situation to extremes, imagine a product consisting of 9500 separate executables, one for every virus…

Needless to say, the manufacturers of products with add-ins do not agree – to some extent, they are right. It would indeed be unfair to imply that they were incapable of handling macro viruses, so the presence and functionality will be discussed in the article, but they will not be used to form part of the headline detection figures. These will remain the sole domain of the main scanner.

There must be an interesting dilemma in the minds of these companies: on one level they cannot wait for the demise of DOS and its puny memory limits and annoyingly restrictive design. The eventual end to DOS’ incredibly long-drawn-out death throes will bring all that to a close, and relegate problems of this nature to distant memory (at least until the limit of the next OS is reached…). On the other hand, DOS was where it all began for anti-virus companies; it will be a shame to see it fade away. Nonetheless, the products will live on, converted to the new generation of operating systems. Plus ça change, plus c’est la même chose?

(The January 1997 comparative review referred to in this article can be read here)



Latest articles:

Cryptojacking on the fly: TeamTNT using NVIDIA drivers to mine cryptocurrency

TeamTNT is known for attacking insecure and vulnerable Kubernetes deployments in order to infiltrate organizations’ dedicated environments and transform them into attack launchpads. In this article Aditya Sood presents a new module introduced by…

Collector-stealer: a Russian origin credential and information extractor

Collector-stealer, a piece of malware of Russian origin, is heavily used on the Internet to exfiltrate sensitive data from end-user systems and store it in its C&C panels. In this article, researchers Aditya K Sood and Rohit Chaturvedi present a 360…

Fighting Fire with Fire

In 1989, Joe Wells encountered his first virus: Jerusalem. He disassembled the virus, and from that moment onward, was intrigued by the properties of these small pieces of self-replicating code. Joe Wells was an expert on computer viruses, was partly…

Run your malicious VBA macros anywhere!

Kurt Natvig wanted to understand whether it’s possible to recompile VBA macros to another language, which could then easily be ‘run’ on any gateway, thus revealing a sample’s true nature in a safe manner. In this article he explains how he recompiled…

Dissecting the design and vulnerabilities in AZORult C&C panels

Aditya K Sood looks at the command-and-control (C&C) design of the AZORult malware, discussing his team's findings related to the C&C design and some security issues they identified during the research.

Bulletin Archive

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.