VB2017 preview: Stuck between a ROC and a hard place

Posted by   Martijn Grooten on   Aug 24, 2017

Authors of security software in general, and anti-virus software in particular, have always needed to find the right balance between a high detection rate and a low false positive rate – something that has become even more important with advances in machine-learning detection technologies. Making the model too strict will result in false positives, while making it too relaxed will result in missed malware – and in both cases unhappy customers.

To help guide the decision as to where to pitch that balance, Microsoft researchers Holly Stewart and Joe Blackbird came up with the interesting idea of using the wisdom of the crowd: they decided to use Microsoft's telemetry to look at what kind of errors (false positives or false negatives) had prompted end-users to switch anti-virus solutions.

In their paper, which Holly will present at VB2017 in Madrid, the researchers discuss what the data showed; they also looked at how different user types, or users in different geographic locations, may respond differently to errors made by their anti-virus solutions. It is an interesting approach that will likely be useful to other security software developers who have to make similar decisions when it comes to applying their machine-learning models.

Receiver Operating Characteristic (ROC) curve that plots the true positive (good detection) rate against the false positive (incorrect detection) rate. 


You can still register for VB2017 to hear more about this research and to learn from the work of more than 50 security researchers from all over the world!




Latest posts:

Standalone product test: FireEye Endpoint

Virus Bulletin ran a standalone test on FireEye's Endpoint Security solution.

VB2017 video: Consequences of bad security in health care

Jelena Milosevic, a nurse with a passion for IT security, is uniquely placed to witness poor security practices in the health care sector, and to fully understand the consequences. Today, we publish the recording of a presentation given by Jelena at…

Vulnerabilities play only a tiny role in the security risks that come with mobile phones

Both bad news (all devices were pwnd) and good news (pwning is increasingly difficult) came from the most recent mobile Pwn2Own competition. But the practical security risks that come with using mobile phones have little to do with vulnerabilities.

VB2017 paper: The (testing) world turned upside down

At VB2017 in Madrid, industry veteran and ESET Senior Research Fellow David Harley presented a paper on the state of security software testing. Today we publish David's paper in both HTML and PDF format.

VB2017 video: Turning Trickbot: decoding an encrypted command-and-control channel

Trickbot, a banking trojan which appeared this year, seems to be a new, more modular, and more extensible malware descendant of the notorious Dyre botnet trojan. At VB2017, Symantec researcher Andrew Brandt presented a walkthrough of a typical…