The (testing) world turned upside down

Thursday 5 October 11:00 - 12:30, Small talks

David Harley (ESET)
John Hawes (AMTSO)



We often hear that anti-virus is dead, but if that is really so, where does it leave anti-malware product testing?

After decades of slow progress, security product testing has been moving away from the chaotic practices of the early 90s, to models of better practice as to some extent codified in the AMTSO 'Fundamental Principles of Testing'. Yet we've recently seen a resurgence in approaches to comparative testing that have long been flagged with a red light:

  • Disabling of layers of functionality and the demotion of whole product testing
  • Simulation as a comparative testing tool
  • Malware creation
  • Opaque sourcing, selection, classification and validation of samples
  • Promotion of D-I-Y testing as superior to independent testing.

Have so many of the assumptions made on both sides of the vendor/tester divide been wrong all along? Or is just this another instance of The (Testing) World Turned Upside Down by marketing?

In this paper, we re-examine those assumptions, set in the context of:

  • The good, the bad and the ugly in early product testing, and the slow-burn reaction of the security industry, culminating in the CARO testing workshop and the first steps towards the foundation of AMTSO.
  • The painful evolution of AMTSO into a source of testing guidelines and somewhat less reliable mediation between the opposed yet interdependent testing and vendor communities.
  • VirusTotal's re-engineering of its policies, and the impact on AMTSO of the subsequent semi-assimilation of self-named 'next-gen' vendors into its membership.
  • A new generation of conflicts between vendors and testers.
  • The claimed divergence in anti-malware technologies and mindset across the spectrum of mainstream and newer vendors. Does this divergence necessitate new testing methodologies? How can such methodologies be appropriately evolved, and how successfully can AMTSO play its part?

Or are both AMTSO and mainstream independent testing doomed to failure and fragmentation?

The presentation of this paper by ESET Senior Reseearch Fellow David Harley will be followed by a discussion among the audience led by AMTSO's John Hawes.



Register.jpg

VB2017 OVERVIEW

CALL FOR LAST-MINUTE PAPERS

WHY ATTEND

SPEAKERS

PROGRAMME

REGISTER NOW!

VENUE

BOOK HOTEL

VB2017 DRINKS RECEPTION

VB2017 FOOSBALL TOURNAMENT

2017 PÉTER SZŐR AWARD


Other VB2017 papers

Walking in your enemy's shadow: when fourth-party collection becomes attribution hell

Juan Andres Guerrero-Saade (Kaspersky Lab)
Costin Raiu (Kaspersky Lab)

Attribution is complicated under the best of circumstances. Sparse attributory indicators and the possibility of overt…

Mariachis and jackpotting: ATM malware from Latin America

Thiago Marques (Kaspersky Lab)

Fabio Assolini (Kaspersky Lab)

Of all the forms of attack against financial institutions in the world, the ones that are most likely to combine traditional…

Keynote address: Inside Cloudbleed

John Graham-Cumming (Cloudflare)

In February 2017, Cloudflare was revealed to have been leaking private information including HTTP headers, cookies and POST data…