Mobile smart phones and tablets provide not only high resolution cameras but also powerful processing capabilities. Yet, these devices consume more energy at the same time. Thus, energy efficiency and battery life have been extremely important topics for mobile devices. Dynamic Voltage and Frequency Scaling (DVFS) has been commonly used to reduce energy consumption for digital systems. In this paper, we propose a novel approach, which integrates DVFS to smart phones performing object tracking. If the processor is continuously run at the highest frequency, this would provide higher frame processing rate, but would consume significantly more energy. If the lowest CPU frequency is used, energy consumption would be decreased, but the frame processing rate would be low. In the proposed approach, instead of running at a certain set frequency, The CPU frequency is adjusted depending on the scene so that energy consumption is decreased without significantly lowering the frame processing rate. The tracking algorithm employed is based on the color-blob tracking provided by OpenCV. All the experiments have been performed on Android smart phones, both Google Nexus S and Samsung Galaxy S3. We measured the real-time energy consumption together with the frame per second processing rate as the metrics. The results show that by introducing DVFS, the proposed method provides 24.29% and 10.3% savings in energy consumption on Nexus S and Galaxy S3, respectively, compared to using the maximum frequency setting.