Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks

Ji Li, Zihao Yuan, Zhe Li, Caiwen Ding, Ao Ren, Qinru Qiu, Jeffrey Draper, Yanzhi Wang

Research output: Chapter in Book/Entry/PoemConference contribution

40 Scopus citations


Recently, Deep Convolutional Neural Networks (DCNNs) have made unprecedented progress, achieving the accuracy close to, or even better than human-level perception in various tasks. There is a timely need to map the latest software DCNNs to application-specific hardware, in order to achieve orders of magnitude improvement in performance, energy efficiency and compactness. Stochastic Computing (SC), as a low-cost alternative to the conventional binary computing paradigm, has the potential to enable massively parallel and highly scalable hardware implementation of DCNNs. One major challenge in SC based DCNNs is designing accurate nonlinear activation functions, which have a significant impact on the network-level accuracy but cannot be implemented accurately by existing SC computing blocks. In this paper, we design and optimize SC based neurons, and we propose highly accurate activation designs for the three most frequently used activation functions in software DCNNs, i.e, hyperbolic tangent, logistic, and rectified linear units. Experimental results on LeNet-5 using MNIST dataset demonstrate that compared with a binary ASIC hardware DCNN, the DCNN with the proposed SC neurons can achieve up to 61X, 151X, and 2X improvement in terms of area, power, and energy, respectively, at the cost of small precision degradation. In addition, the SC approach achieves up to 21X and 41X of the area, 41X and 72X of the power, and 198200X and 96443X of the energy, compared with CPU and GPU approaches, respectively, while the error is increased by less than 3.07%. ReLU activation is suggested for future SC based DCNNs considering its superior performance under a small bit stream length.

Original languageEnglish (US)
Title of host publication2017 International Joint Conference on Neural Networks, IJCNN 2017 - Proceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Number of pages7
ISBN (Electronic)9781509061815
StatePublished - Jun 30 2017
Event2017 International Joint Conference on Neural Networks, IJCNN 2017 - Anchorage, United States
Duration: May 14 2017May 19 2017

Publication series

NameProceedings of the International Joint Conference on Neural Networks


Other2017 International Joint Conference on Neural Networks, IJCNN 2017
Country/TerritoryUnited States


  • Activation Function
  • Deep Convolutional Neural Networks
  • Deep Learning
  • Stochastic Computing

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence


Dive into the research topics of 'Hardware-driven nonlinear activation for stochastic computing based deep convolutional neural networks'. Together they form a unique fingerprint.

Cite this