An Energy-Efficient Online-Learning Stochastic Computational Deep Belief Network

Yidong Liu, Yanzhi Wang, Fabrizio Lombardi, Jie Han

Research output: Contribution to journalArticle

7 Scopus citations

Abstract

Deep neural networks (DNNs) are effective machine learning models to solve a large class of recognition problems, including the classification of nonlinearly separable patterns. The training of DNNs is, however, particularly difficult due to the large size and high energy consumption of the networks. Recently, stochastic computation (SC) has been considered to implement DNNs to reduce the hardware cost. However, it requires a large number of random number generators (RNGs) and long stochastic sequences that lower the energy efficiency of the network. To overcome these limitations, we propose the design of an energy-efficient deep belief network (DBN) with online learning capacity based on stochastic computation. In the SC-DBN, a reconfigurable structure is utilized to implement the fast greedy learning algorithm and an adaptive moment estimation (ADAM) circuit is designed to improve the speed of the training process. An approximate SC activation unit (A-SCAU) is further designed to implement different types of activation functions in the neurons. The A-SCAU is immune to signal correlations, so the RNGs can be shared among all neurons in the same layer with no accuracy loss. The area and energy of the proposed design are less than 5.5% and 3.7% (or 29.3% and 33.3%) of a pipelined 32-bit floating-point (or an 8-bit fixed-point) implementation. The proposed SC-DBN design achieves a higher classification accuracy compared with the fixed-point implementation. The accuracy is in a range of 0.12% to 0.37% lower than the floating-point design with a significantly lower (or slightly higher) energy consumption than the pipelined (or non-pipelined) circuit for both online learning and inference processes.

Original languageEnglish (US)
Article number8403221
Pages (from-to)454-465
Number of pages12
JournalIEEE Journal on Emerging and Selected Topics in Circuits and Systems
Volume8
Issue number3
DOIs
StatePublished - Sep 1 2018

Keywords

  • cognitive computing
  • deep belief network
  • rectifier linear unit
  • Stochastic computing

ASJC Scopus subject areas

  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'An Energy-Efficient Online-Learning Stochastic Computational Deep Belief Network'. Together they form a unique fingerprint.

  • Cite this