Dynamical system theory for the detection of anomalous behavior in computer programs

Code injection is a common approach which is utilized to exploit applications. We introduce some of the well-established techniques and formalisms of dynamical system theory into analysis of program behavior via system calls to detect code injections into an applications execution space. We accept a program as a blackbox dynamical system whose internals are not known, but whose output we can observe. The blackbox system observable in our model is the system calls the program makes. The collected system calls are treated as signals which are used to reconstruct the system's phase space. Then, by using the well-established techniques from dynamical system theory, we quantify the amount of complexity of the system's (program's) behavior. The change in the behavior of a compromised system is detected as anomalous behavior compared with the baseline established from a clean program. We test the proposed approach against DARPA-98 dataset and a real-world exploit and present code injection experiments to show the applicability of our approach.