The efficiency of thermal insulation of buildings has direct impact on power consumption for heating and ventilation. Thermal leaks are critical defects on building insulation. Infrared sensitive handheld thermal cameras are commonly used to perform manual inspection of buildings to detect thermal leaks by human experts. In this paper, we propose an algorithm that autonomously detects heat leakages from thermal images of building structures. The target platform for the proposed algorithm is an unmanned aerial vehicle (UAV) that collects thermal images of buildings, and detects heat leakages autonomously. Using UAVs, as opposed to the current practice of doing manual inspection, not only speeds up the process, but also provides the capability of capturing data from hard to reach places, such as roofs, in a more convenient and safe manner. For evaluation, thermal images have been captured both from inside, by a handheld camera, and outside. The outside thermal images have been captured by either a handheld camera or an IR camera installed on a UAV. Experimental results show that the proposed approach is very promising with overall precision and recall rates of 91% and 89%, respectively on handheld dataset, and 34% and 77%, respectively on the UAV dataset.