Research paper shows it may be possible to distinguish malware traffic using TLS

Posted by   Martijn Grooten on   Jun 22, 2017

Researchers at Cisco have published a paper (PDF) describing how it may be possible to use machine learning to distinguish malware command-and-control (C&C) traffic using TLS from regular enterprise traffic, and to classify malware families based on their encrypted C&C traffic.

The need for malware to communicate with its operators, so that it can receive instructions and exfiltrate information from infected systems, is a weak point – it can't easily hide its activity from security products scanning network traffic. For this reason, the trend among malware of using SSL/TLS – the protocol over which a significant portion of today's web and email traffic is sent – is an understandable one.

A good encryption protocol makes encrypted content indistinguishable from random noise, but while TLS uses top-class encryption standards, it cannot avoid the use of metadata that can give away some essential details of the communication.

Even if one ignores the remote IP address and the domain sent in the certificate, both of which can help detect a known malware family, TLS includes explicit metadata, such as the cipher suites and TLS extensions offered and used, as well as more implicit metadata, such as the length and frequency of the packets and the variation seen in them.

The Cisco researchers trained their machine-learning classifier using a combination of malicious TLS traffic and legitimate enterprise TLS traffic. The classifier was able to identify the TLS traffic of most malware families with high accuracy – even that of families that had not been present in the training set.

tlsmalwareindicators.png

The research is very much a work-in-progress and, as befits a good research paper, its authors openly admit the limitations to their work. For instance, the malware was run in Windows XP-based sandboxes, which could have helped the detection: malware often inherits TLS properties from the operating system in which it runs. At the same time, malware is mostly likely to live on older operating systems, making this set-up not too different from a real-world scenario.

It is also important to note that the classifier was not able to say anything about the content of the traffic; it would thus be useless as part of a data-loss prevention system. TLS, especially its most recent versions, is one of the strongest Internet protocols, and the fact that it properly protects  content is a very good thing, even if it can be frustrating for malware analysis and detection.

At Virus Bulletin, we have repeatedly shown how malicious web traffic can be blocked by security products. Organizations using a web security gateway will have to make a decision as to whether to have it inspect TLS-encrypted web traffic as well. While I think that, in most scenarios, inspecting the traffic is a compromise worth making, this research shows that one may be able to block malware's ability to connect to its owners without being able to decrypt the traffic.

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest posts:

VB2018 paper: The modality of mortality in domain names

Domains play a crucial role in most cyber attacks, from the very advanced to the very mundane. Today, we publish a VB2018 paper by Paul Vixie (Farsight Security) who undertook the first systematic study into the lifetimes of newly registered domains.…

VB2018 paper: Analysing compiled binaries using logic

Constraint programming is a lesser-known technique that is becoming increasingly popular among malware analysts. In a paper presented at VB2018 Thaís Moreira Hamasaki presented an overview of the technique and explained how it can be applied to the…

Virus Bulletin encourages experienced speakers and newcomers alike to submit proposals for VB2019

With a little less than a month before the deadline of the call for papers for VB2019, Virus Bulletin encourages submissions from experienced speakers and newcomers alike.

VB2018 paper: Internet balkanization: why are we raising borders online?

At VB2018 in Montreal, Ixia researcher Stefan Tanase presented a thought-provoking paper on the current state of the Internet and the worrying tendency towards raising borders and restricting the flow of information. Today we publish both his paper…

The malspam security products miss: banking and email phishing, Emotet and Bushaloader

The set-up of the VBSpam test lab gives us a unique insight into the kinds of emails that are more likely to bypass email filters. This week we look at the malspam that was missed: banking and email phishing, Emotet and Bushaloader.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.