VB100 comparative review on SUSE Linux Enterprise Server 11

2013-02-22

John Hawes

Virus Bulletin
Editor: Helen Martin

Abstract

This month sees the VB test team taking their annual look at anti-malware products for the Linux world. John Hawes has the details.


Introduction

Our annual departure from the familiar Windows fare to the rather more exotic shores of the Linux world always has both upsides and downsides from a testing point of view. On the one hand, the market is significantly less crowded, and whereas a Windows desktop comparative would regularly feature 50 or more products, in Linux month it would be a surprise to see more than a dozen or so. Having fewer products to deal with should mean far less time is required to run through the testing programme, and the simplicity of automating tasks on the platform should also cut out a hefty chunk of hands-on time. On the other hand, products for Linux tend to be considerably less user-friendly than their Windows counterparts, with esoteric processes for installation and configuration noted frequently in previous Linux comparatives.

Starting late thanks to end-of-year holidays, products were taken in on the 19 December deadline, but testing work did not commence until well into January, with the bulk of it finished by early February.

In the past, a smaller field of competitors has often tended to go hand in hand with a higher VB100 pass rate. With only the most experienced of companies producing Linux products (or at least entering them in our comparatives), Linux tests tend to elicit fewer problems than usual in the certification areas. With only minimal expansion of our false positive sets this month, and relatively few tricky items existing in the WildList sets, the chances of seeing a 100% pass rate seemed fairly high – something which has not happened in at least the 40 comparatives run since testing duties fell into my hands in 2006, and as far as I can tell not at all in the history of the VB100. Of course, there was always the chance of a surprise upset, so as ever we knuckled down to testing with a mixture of anticipation and trepidation.

Platform and test sets

The latest version of Novell’s SUSE Linux Enterprise Server platform, version 11 SP2, had a few surprises for us, having undergone a major kernel version change since its last appearance (already on v11) in the February 2010 comparative. Despite being almost a year old, this update promised to cause some havoc as, according to our contacts, the new kernel effectively broke the dazuko/redirfs approach (usually one of the most popular techniques) to providing on-access protection on Linux. In previous comparatives we have prepared and installed the dazuko module onto our systems from the off, but in this case we decided not to, and instead to see what options were provided by the products at submission time. So we went for a fairly bare install of the platform, which proved a relatively straightforward process. The GNOME desktop is installed by default – we left this as it was, expecting many products to provide some sort of interface. Setting up networking was a fairly routine task, with one system left running Windows XP SP3 and connected to network shares on each test machine to act as a client for the on-access tests.

In a slight change to our traditional methodology, the WildList cut‑off was set for the same date as the product submission deadline, 19 December. As our new approach means that testing against the WildList does not commence for some time after the submission deadline, and uses latest updates at the time of testing, it seemed reasonable to move the cut‑off point forward a little to ensure we’re keeping everyone well and truly on their toes. The November WildList, posted just a couple of days before our deadline, was thus used as the basis of our main certification sets.

The clean sets were little changed this month, with a relatively small number of new items added, mostly focusing on business software as befits the business leaning of this month’s products. The RAP test sets were compiled using the usual approach, with the final sets averaging 25,000 samples per week. The systems for building our Response sets were also unchanged, producing sets of around 2,500 samples per day.

The biggest changes were in our speed and performance sets, where a dedicated set of Linux files was built for speed measures. This contained the contents of several important sections of a standard Linux installation, including the /bin, /sbin, /etc and /lib directories. Some adjustments had to be made to our measurement processes to fit in with the systems, and this meant dropping our ‘standard activities’ measure for this test, which was unsuitable for the client/server approach. Otherwise things remained much the same as usual.

Most of the expected hard core of regulars turned up on the product deadline, with a couple of the larger companies – sporadic at best in their appearances of late – missing once again. The recent demise of VirusBuster meant that a couple of products we might usually expect to see were absent, and with a few more failing to appear – either thanks to issues with the platform or to internal company reorganizations – just nine products made the final cut, equalling the number seen in the last SUSE test three years ago.

Results

Avast Software avast! For Linux/Unix

Main version: v3.2.1

Update versions: 121219-1, 130121-1, 130123-1, 130128-0

Avast’s developers were among the first to warn us of potential problems with the dazuko subsystem, but once we informed them that we were happy to change the kernel in use, everyone was happy. The submission was sent in as a 160MB archive, which contained everything needed to roll back the kernel and install the dazuko and redirfs modules, as well as the three RPM packages comprising the product itself. These were installed using the usual method, and everything else was found to be fairly standard, much to our satisfaction: configuration was adjusted using plain files in /etc, the location of the main executables was added to the search path, and ample man pages were provided describing the usage of the product in clear and simple terms. The only action required initially was the adjustment of one config file to point to a licence key, and then everything was up and running. The whole process, including the kernel adjustments and the reboot this necessitated, took only a couple of minutes. Updates were fairly speedy too, rarely taking more than 30 seconds or so.

The product ran without issues, proving just as stable to run as it was simple to operate. Scanning speeds were not bad – very fast indeed over our sets of media and miscellaneous files – with overheads showing a similar pattern. Memory use was low, but CPU use fairly high during busy times, not surprisingly. Detection was decent too – looking a little below par in some of our graphs, but this is entirely thanks to the quality of the opposition. Detection in the Response sets started very respectably and showed just a slight downward trend into the more recent days. The RAP sets were mostly well covered, but a rather sharp blip in the oldest week brought the average down somewhat.

The clean sets brought no surprises, and the WildList sets seemed to be well covered too, until the third of three test runs showed a loss of detection for a pair of samples in the standard set. The issue remained during further checks over the next few days, and was reported to the developers as a matter of urgency. Their feedback implied that the problem was thanks to the aged nature of the Linux product, lagging several cycles behind its Windows cousins (which were unlikely to be affected by the issue). Nevertheless, despite a generally strong showing Avast doesn’t quite make the grade for VB100 certification, putting paid to our hopes for a clean sweep before we’d really got started. Our test history for Avast now shows one fail and five passes in the last six tests; ten passes and two fails in the last two years. Stability was fine this month once again, with a ‘Solid’ rating easily earned.

ItW Std: 99.88%

ItW Std (o/a): 99.88%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Solid

AVG 2012 Linux Server Edition

Main version: 12.0.1795

Update versions: 2637/5465, 2639/5549, 2639/5553, 2639/5565

Clearly having received some recent attention from its developers, AVG’s product managed to circumvent the problems with dazuko, which remains supported where appropriate but on more recent kernels the fanotify system has been implemented. Installation of the product was fairly straightforward, with a single RPM of just over 100MB provided. This put everything in place nice and simply, but configuration was a little more complex than might be desirable, with a fairly lengthy syntax required to pass even fairly minor changes into the product. Figuring out the right wording was by no means straightforward, but was helped by some comprehensive man pages, and things were up and running in a decent amount of time. Initial updates seemed rather start-stop, taking over eight minutes to run through on the first attempt, but later runs were much smoother, all completing in under a minute.

Operation was generally fairly simple once the initial set up had been completed, and things ran without problems. Scanning speeds were pretty good, and overheads not bad at all either, with a fair amount of RAM use but pretty low use of CPU cycles. Detection was excellent, with consistently high rates throughout our Response sets and RAP scores also starting very high indeed, tailing off rather more than most into the proactive week, but remaining pretty decent.

There were no problems in the core sets and a VB100 is comfortably earned by AVG; the vendor stands at one fail and five passes in the last six tests; two fails and ten passes in the last two years. This was another very smooth run, meriting a ‘Solid’ rating.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Solid

Bitdefender Security for Samba Servers

Main version: 3.1.2

Update versions: N/A

Bitdefender’s Samba-focused product also came as a single item, an executable-wrapped RPM installer which set things up simply and quickly, with minimal extra work needed to get everything working nicely. Updates were swift too, but the design shies away from standard practice, using a complex system of commands passed into the main executable to make adjustments rather than the more common process of editing a plain text configuration file. Nevertheless, with a little trial and error it soon became reasonably usable. Help was provided in the form of a set of man pages, but the set up did not appear to have functioned properly so these were not discovered until late in the day, when rummaging through the product’s install directory for some ideas. The main executable was not added to the search path automatically, either.

On-access protection covers Samba shares only, by means of a VFS object pointed to by some tweaks to the Samba configuration file, which are put in place automatically during the set up. Logging was a little verbose but clear enough and pretty reliable, while the basic syntax of the scanner was fairly straightforward once the schema had been worked out. Scanning speeds were on the slow side, and on access overheads perhaps a little higher than some this month, with fairly high use of RAM but low CPU use.

Detection was superb though, continuing a recent string of excellent scores from Bitdefender, with a distinct but fairly minor decline through the days of the Response sets and RAP numbers consistently excellent, even the proactive set very well covered. The certification sets caused no problems, and a VB100 award is easily earned, maintaining Bitdefender’s recent run of success and keeping it on 12 passes in the last two years of comparatives. No serious problems were noted, but the failure of the set up of the man pages was just enough to affect our scoring, meaning only a ‘Stable’ rating is earned.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Stable

eScan for Linux File Server

Main version: 5.6.1

Update versions: N/A

The developers of eScan provided a fairly comprehensive submission: the four main RPM packages, weighing in at 190MB, were accompanied by a licence file, a dependency list and a PDF user guide. The dependency list proved unnecessary, with all of the requirements already in place on a fairly basic system, and the user guide was only really needed to confirm the order in which the RPMs should be run, and to provide the syntax for changes needed in the Samba configuration file. Man pages were provided for the main executable components, but not for the daemons, however configuration was mostly performed via a browser-based GUI, which proved simple to access and operate, providing ample information and controls for most purposes. Updates were fairly speedy, completing in little over a minute on most runs, although the ClamAV-based part regularly reported issues which could only be resolved by updating core components manually; the Bitdefender part of the product operated smoothly throughout.

Speeds were not too bad and overheads just a little high, with RAM use around average and CPU use fairly low. Detection scores, as expected, were excellent just about everywhere, showing slight declines into the most recent sets but from very high starting points. No issues were encountered in the core sets, earning eScan a VB100 award and putting it on a perfect 12 passes in the last dozen tests. With no stability issues to report, a ‘Solid’ rating is earned.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Solid

ESET Security for Linux

Main version: 4.0.5

Update versions: Scanner 7818, 7919, 7926, 7943

ESET’s submission was a nice and compact 60MB single RPM installer, which ran through quickly and simply; final set up of the product included adjustment of just a single line in the Samba resource control script and a few tweaks to the product’s comprehensive and clearly documented configuration file. Support is also provided for dazuko protection if required. Updates were rapid at around two minutes on average.

Operation is simple and conforms to standard practices, with clear and thorough man pages detailing all components and everything positioned more or less where one would expect to find it. Scanning speeds were decent for the most part, blazing fast over our media sets, and overheads were very low indeed, although we could find no option which would effectively activate archive scanning on access. RAM and CPU use were both around average, and detection was solid, with very consistent and highly respectable scores through the Response sets, a good start in the RAP sets and not too steep a decline into the latter weeks.

The core sets presented no problems as usual, and ESET’s epic string of successes continues, with no tests failed or even skipped for many years. No stability issues were encountered, earning the product another ‘Solid’ rating.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Solid

F-Secure Linux Security

Main version: 9.20.2520

Update versions: 2012-12-19_07, 2013-01-22_06, 2013 01-24_06, 2013-01-29_04

The F-Secure product was provided as a slimline 66MB installer, although this lacked updates and had to be installed on the deadline day to take snapshots of the latest definitions for the RAP tests. The set up process involved unpacking an archive and running a script contained within it. This seemed to run smoothly, with no major alerts, but on closer scrutiny of the scrolling output, some errors were observed in compiling the dazuko and redirfs modules. Knowing these were not compatible with the kernel in use, we set up a second system with an older kernel. Here, there were no issues and on-access protection operated well, but on the original machine the product’s browser-based GUI seemed unaware of any installation issues, insisting that protection was fully functional. We also saw some error messages concerning an alternative GNOME desktop interface, which seemed unable to run properly.

With the initial confusion ironed out (the developers informed us that the platform version we were testing on is not officially supported, which makes the problems we noted rather less severe), things moved along nicely. The GUI provided decent levels of information and configuration, and the syntax for the command-line scanner was clear and sensible. Scanning speeds were a little underwhelming, but overheads were mostly decent, although a few sets did slow down considerably when the settings were turned up to the most thorough. Use of memory was very low throughout testing, but CPU use was well up at busy times.

Detection was solid, as we would expect from the Bitdefender engine included in the product alongside the company’s own work, with splendid scores everywhere. The core sets were brushed aside comfortably, and a VB100 award is earned. F-Secure has a somewhat erratic pattern of submissions of late, and now stands on three passes and one fail from four entries in the last six tests; eight passes and just that single fail in the last two years. There were a few wobbles observed, including the erroneous assertion that protection was in place when it was not, meaning a stability rating of no more than ‘Fair’.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Fair

Kaspersky Anti-Virus 8.0 for Linux File Servers

Main version: 8.0.2-160

Update versions: N/A

Of all the products in this month’s line up, Kaspersky’s inspired the most dread among the test team thanks to difficulties encountered in previous Linux tests. The initial set up, from a single 57MB RPM file, was fairly painless and speedy, with a set up script provided to do the work of compiling and implementing on-access protection, licensing and so on. The on access component appears to be a custom offering loosely based on the redirfs approach, but there were no problems with getting it set up, and options to protect Samba shares explicitly were also provided. Once these first stages were complete, the work of figuring out the configuration and operation process began in earnest, first of all to apply the updates downloaded separately for the RAP tests. This involved lengthy perusal of an epic 178-page user manual (no man pages are provided), and considerable trial and error as the details of adjusting the update process were less than perfectly clear. With no configuration file as such, all changes to settings had to be performed by getting the product to dump the settings of the module in question to a file, tweaking that file and then reading it back in again. In most cases this took some time and a fair amount of brain power. After a while it all became reasonably intuitive, once the basic processes were memorized, but mistakes still occurred disappointingly frequently, meaning testing took far longer than for any of the other products.

In addition, we had some problems getting through the RAP tests, with scans crashing out with errors which hinted at problems in processing log data. This was confirmed by the developers, who provided an alternative syntax to the scan command. This proved more effective, but only after much extra work had been put into trying to nurse the scanner through the sets. In the end, we saw some fairly slow scanning speeds, especially over our set of archives (which were scanned in-depth by default) and the set of Linux files, which included a large number of small files. On access overheads were excellent though, with low measures across the board, and archive handling still very reasonable when settings were turned right up. RAM use was around average, but CPU use distinctly high.

Detection rates were very good, remaining fairly steady through the Response sets, actually improving in the latter parts, and starting very well in the RAP sets, although dropping quite sharply into the proactive week. The core sets were dealt with properly, and a VB100 award is well deserved, putting Kaspersky on six passes in the last six tests; longer term things are a little more shaky, with nine passes and two fails in the last two years. There were a few fairly minor problems this month, which only occurred under heavy stress, earning the product a ‘Stable’ rating.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Stable

Norman Endpoint Protection for Linux

Main version: 7.0.20

Update versions: 16375682, 17218629, 17285786, 17472528

Norman’s Linux solution was submitted as a tiny 6.4MB main installer, with 170MB of offline updates for the RAP sets. This hefty update size was reflected in standard installs, which seemed to complete very rapidly but required considerably more time doing background set up tasks, much of which seemed to revolve around downloading and implementing updates. Indeed, on at least a couple of installs, the product remained non functional even after leaving it overnight and giving the entire system a reboot. Most proved reasonably successful after a decent wait though. Configuration was mostly done through another browser GUI, which closely resembled the company’s Windows products, making for a pleasant continuity of usage patterns (at least when the GUI could be persuaded to open properly).

On-access protection is provided using an unusual FUSE based approach, which worked fine with the default layout, but we were unable to extend this to cover additional areas. As a result, all our on access tests had to be re-purposed to operate in the /home directory – the only area which seemed to be protected. On demand scans were a little slow over archives, Windows binaries and Linux files, but reasonable elsewhere. On access overheads, on the other hand, were fairly heavy for the most part but huge over the set of binary files, with the product’s sandbox component clearly adding heavily to the time required to mark files as safe. Some sort of caching of recent results appears to be needed, as much time was spent re-analysing items that had been checked only a few minutes earlier. RAM use was fairly high, and CPU use a little above average too.

Detection was very good though, at least over the older sets, with a fairly sharp drop in the second halves of both the Response and RAP tests. The core sets presented no issues however, and Norman earns a VB100 award. The company now boasts a fairly steady five passes and one fail in the last six tests; ten passes and two fails in the last two years. There were a number of install problems this month, most significantly with trying to apply on-access protection to non-default areas, meaning only a ‘Fair’ stability rating.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Fair

Sophos Anti-Virus Protection for Linux

Main version: 4.84.0

Update versions: 2.03.038/3.38.1/4.84, 3.39.0/4.85

The last of this month’s rather slender field, Sophos provided its solution as a 255MB bundle, with separate updates of 6MB. The install runs from a script inside the main archive package, which ran through the set up, licensing and initial configuration in a clear and simple manner. On access protection from the company’s own ‘Talpa’ service compiled automatically and without problems, and updates were zippy after some initial confusion over an expired licence code. A web admin tool is available but mainly provides information, with most configuration performed by passing commands into the main control binary – a process which is a little fiddly, but reasonably easy to work out thanks to the copious man pages provided. One adjustment, disabling the ‘live protection’ cloud look up system for the RAP tests, could not be found in the official processes but was simple enough to do by editing an XML config file.

Scanning was very fast indeed, at least with the default settings, slowing down considerably when settings were extended to cover all file and archive types. Overheads were very light, although it should be noted that the on-access mode is limited to exclude archives. RAM use was average and CPU use very low.

Detection was a little below par in the RAP sets, looking further behind the rest than one might expect thanks to an exceptionally strong field this month, but in the Response sets scores were excellent, demonstrating the effectiveness of the cloud system which should be in place in most cases. The WildList and clean sets were handled without incident, and Sophos is a worthy winner of another VB100 award, putting it on five passes and one fail in the last six tests; ten passes and two fails in the last two years. Stability was excellent this month, earning a ‘Solid’ rating.

ItW Std: 100.00%

ItW Std (o/a): 100.00%

ItW Extd: 100.00%

ItW Extd (o/a): 100.00%

False positives: 0

Stability: Solid

Results tables

(Click for a larger version of the table)

(Click for a larger version of the table)

(Click for a larger version of the table)

On-demand throughput graph.

(Click for a larger version of the table)

File access lag time graph.

(Click for a larger version of the table)

Response graph.

(Click for a larger version of the table)

Performance measures graph.

(Click for a larger version of the table)

(Click for a larger version of the table)

(Click for a larger version of the chart)

Conclusions

Not quite the clean sweep we had been hoping for, but a pretty near miss, with only one product failing to make the grade (and that was only in one of the three test runs). False positives were absent (perhaps in part thanks to a less extensive than usual update to our clean sets), and the WildList sets were for the most part handled excellently, with just that single momentary lapse from one of the participants.

Otherwise, we saw pretty much what we had expected from the available Linux products, some sticking with very simple, standard approaches, others inventing new wheels in bewildering shapes and flavours. Many firms seem content to let their Linux solutions slide along with minimal maintenance, while others have clearly invested in bringing them into the 21st century, with all sorts of centralized administration gizmos and multi-function protective layers included.

Stability was generally very impressive, with far fewer problems endured than we would normally expect in a Windows test, although whether this is due to the generally higher quality of the participating vendors here, or to the relative simplicity of most of the products, is difficult to judge. Next time we will return to the evergreen Windows XP, and expect to see not only a far wider field of products taking part but also a far wider range of issues to vex us.

Technical details

Test environment. All products were tested on identical machines with AMD Phenom II X2 550 processors, 4GB RAM, dual 80GB and 1TB hard drives, running Novell SUSE Linux Enterprise Server 11 SP2, x64 edition. On-access tests were run from a client system running Microsoft Windows XP SP3 (32-bit), on the same hardware specification. For the full testing methodology see http://www.virusbtn.com/vb100/about/methodology.xml.

Any developers interested in submitting products for VB's comparative reviews, or anyone with any comments or suggestions on the test methodology, should contact [email protected]. The current schedule for the publication of VB comparative reviews can be found at http://www.virusbtn.com/vb100/about/schedule.xml.

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest reviews:

VBSpam Comparative Review

In the Q4 2021 VBSpam test the performance of nine full email security solutions, one custom configured solution, one open-source solution and one blocklist was measured against various streams of wanted, unwanted and malicious emails.

VBSpam Comparative Review

In this test the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist was measured against various streams of wanted, unwanted and malicious emails.

VBSpam Comparative Review - June 2021

In this test the performance of eight full email security solutions, one custom configured solution, one open-source solution and one blocklist was measured against various streams of wanted, unwanted and malicious emails.

VB100 Certification Report - April 2021

This report details the VB100 certification results of 38 anti-malware products from 33 different vendors tested during March and April 2021.

VBSpam Comparative Review - March 2021

The performance of 28 email security products was measured against various streams of wanted, unwanted and malicious emails. The results for the 10 products that elected to be tested publicly are included in this report.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.