Say hi to malware - using a deep learning method to understand malicious traffic

Friday 6 October 11:30 - 12:00, Red room

Zhaoyan Xu (Palo Alto Networks)
Tongbo Luo (Palo Alto Networks)
Wei Xu (Palo Alto Networks)
Kyle Sanders (Palo Alto Networks)
Xin Ouyang (Palo Alto Networks)

Recently, thanks to the exponential growth of data size in our daily communications, it has become more and more challenging for security practitioners to identify little drops of malicious traffic among the sea of benign data. In particular, current advanced persistent threat (APT) attacks commonly spread their communication across multiple independent network sessions, making it hard for the traditional IPS signature generation scheme to succeed. Similar to learning a foreign language, without understanding the syntax, semantics and context of malicious communications, it is almost impossible to defend against them.

In this paper, we attempt to take a deep look into the cross-session communication of malware, and to understand their language automatically. To achieve this, we employ multiple deep learning methods to systematically analyse the syntax, semantics and contextual information of the malware's communication. In detail, we split each malware family's network communication into words, packets, and sessions. Then we develop a multi-layer recurrent neural network to describe the internal logics of each malware dialect. Based on the learned model, we can generate highly effective intrusion prevention signatures without any manual effort. Furthermore, we have developed a deep enforcement learning method to handle the variants of malicious traffic. Our method can automatically generate signatures for over 40 malware families. By evaluating millions of live traffic data, our signatures can detect malicious traffic without any false alarms.



Zhaoyan Xu

Zhaoyan Xu is a research engineer at Palo Alto Networks in CA, United States. He joined Palo Alto Networks in 2014 and worked in the area of Internet security. He earned his Ph.D. degree at Texas A&M University, College Station in 2014. His research interests include web security, malware analysis, detection and system security.



Tongbo Luo

Tongbo Luo is a principle security researcher at Palo Alto Networks. His current research interests include cybersecurity, mobile security and security data analysis. He obtained his M.S. and Ph.D. degrees in computer science from Syracuse University in 2014. He is active  in mobile security, cybersecurity, IoT security and applied machine learning for security problems.



Wei Xu

Wei Xu is a security researcher at Palo Alto Networks. His current research interests include web security, network security and security data analysis. His past research works have been published in both academic and industry circles. He was a speaker at VB2012/2014/2015 and Blackhat 2013. He received his B.S and M.S. degrees in electrical engineering from Tsinghua University, Beijing,China, in 2005 and 2007 respectively. He obtained his Ph.D degree in computer science from Penn State University in 2013.   


Kyle Sanders

Kyle Sanders has worked in the IT industry for the last 11 years and is currently the team lead for malware research at Palo Alto Networks. His research interests are in automated malware detection, network forensics and code analysis.



Xin Ouyang

Xin Ouyang is a senior manager at Palo Alto Networks. His current research interests include intrusion detection and prevention systems, web security, security data analysis, and security of the Internet of Things.






Other VB2017 papers

XAgent: APT28 cyber espionage on macOS

Tiberius Axinte (Bitdefender)

This paper provides an in-depth analysis of the macOS version of the APT28 component known as XAgent. We will dissect the…

The state of cybersecurity in Africa: Kenya

Tyrus Kamau (Euclid Consultancy)

The cyber threats Kenya faces range from basic hacking such as website defacements, financial fraud, social media account…

Walking in your enemy's shadow: when fourth-party collection becomes attribution hell

Juan Andres Guerrero-Saade (Kaspersky Lab)
Costin Raiu (Kaspersky Lab)

Attribution is complicated under the best of circumstances. Sparse attributory indicators and the possibility of overt…

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.