HookScout: Proactive binary-centric hook detection

Heng Yin, Pongsin Poosankam, Steve Hanna, Dawn Song

Research output: Chapter in Book/Entry/PoemConference contribution

24 Scopus citations


In order to obtain and maintain control, kernel malware usually makes persistent control flow modifications (i.e., installing hooks). To avoid detection, malware developers have started to target function pointers in kernel data structures, especially those dynamically allocated from heaps and memory pools. Function pointer modification is stealthy and the attack surface is large; thus, this type of attacks is appealing to malware developers. In this paper, we first conduct a systematic study of this problem, and show that the attack surface is vast, with over 18,000 function pointers (most of them long-lived) existing within the Windows kernel. Moreover, to demonstrate this threat is realistic for closed-source operating systems, we implement two new attacks for Windows by exploiting two function pointers individually. Then, we propose a new proactive hook detection technique, and develop a prototype, called HookScout. Our approach is binary-centric, and thus can generate hook detection policy without access to the OS kernel source code. Our approach is also context-sensitive, and thus can deal with polymorphic data structures. We evaluated HookScout with a set of rootkits which use advanced hooking techniques and show that it detects all of the stealth techniques utilized (including our new attacks). Additionally, we show that our approach is easily deployable, has wide coverage and minimal performance overhead.

Original languageEnglish (US)
Title of host publicationDetection of Intrusions and Malware, and Vulnerability Assessment - 7th International Conference, DIMVA 2010, Proceedings
Number of pages20
StatePublished - 2010
Event7th GI International Conference on Detection of Intrusions and Malware and Vulnerability Assessment, DIMVA 2010 - Bonn, Germany
Duration: Jul 8 2010Jul 9 2010

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume6201 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Other7th GI International Conference on Detection of Intrusions and Malware and Vulnerability Assessment, DIMVA 2010

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'HookScout: Proactive binary-centric hook detection'. Together they form a unique fingerprint.

Cite this