Dynamical system theory for the detection of anomalous behavior in computer programs

Nitin Kanaskar, Remzi Seker, Jiang Bian, Vir V. Phoha

Research output: Contribution to journalArticlepeer-review

1 Scopus citations


Code injection is a common approach which is utilized to exploit applications. We introduce some of the well-established techniques and formalisms of dynamical system theory into analysis of program behavior via system calls to detect code injections into an applications execution space. We accept a program as a blackbox dynamical system whose internals are not known, but whose output we can observe. The blackbox system observable in our model is the system calls the program makes. The collected system calls are treated as signals which are used to reconstruct the system's phase space. Then, by using the well-established techniques from dynamical system theory, we quantify the amount of complexity of the system's (program's) behavior. The change in the behavior of a compromised system is detected as anomalous behavior compared with the baseline established from a clean program. We test the proposed approach against DARPA-98 dataset and a real-world exploit and present code injection experiments to show the applicability of our approach.

Original languageEnglish (US)
Article number6392448
Pages (from-to)1579-1589
Number of pages11
JournalIEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews
Issue number6
StatePublished - 2012
Externally publishedYes


  • Anomalous behavior
  • approximate entropy
  • central tendency measure (CTM) dynamical system
  • intrusion detection
  • percent determinism
  • percent ratio
  • percent recurrence
  • recurrence plots
  • system call sequence

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Dynamical system theory for the detection of anomalous behavior in computer programs'. Together they form a unique fingerprint.

Cite this