DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks

Zhe Li, Ao Ren, Ji Li, Qinru Qiu, Yanzhi Wang, Bo Yuan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

27 Citations (Scopus)

Abstract

Deep Convolutional Neural Networks (DCNN), a branch of Deep Neural Networks which use the deep graph with multiple processing layers, enables the convolutional model to finely abstract the high-level features behind an image. Large-scale applications using DCNN mainly operate in high-performance server clusters, GPUs or FPGA clusters; it is restricted to extend the applications onto mobile/wearable devices and Internet-of-Things (IoT) entities due to high power/energy consumption. Stochastic Computing is a promising method to overcome this shortcoming used in specific hardware-based systems. Many complex arithmetic operations can be implemented with very simple hardware logic in the SC framework, which alleviates the extensive computation complexity. The exploration of network-wise optimization and the revision of network structure with respect to stochastic computing based hardware design have not been discussed in previous work. In this paper, we investigate Deep Stochastic Convolutional Neural Network (DSCNN) for DCNN using stochastic computing. The essential calculation components using SC are designed and evaluated. We propose a joint optimization method to collaborate components guaranteeing a high calculation accuracy in each stage of the network. The structure of original DSCNN is revised to accommodate SC hardware design's simplicity. Experimental Results show that as opposed to software inspired feature extraction block in DSCNN, an optimized hardware oriented feature extraction block achieves as higher as 59.27% calculation precision. And the optimized DSCNN can achieve only 3.48% network test error rate compared to 27.83% for baseline DSCNN using software inspired feature extraction block.

Original languageEnglish (US)
Title of host publicationProceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages678-681
Number of pages4
ISBN (Electronic)9781509051427
DOIs
StatePublished - Nov 22 2016
Event34th IEEE International Conference on Computer Design, ICCD 2016 - Scottsdale, United States
Duration: Oct 2 2016Oct 5 2016

Other

Other34th IEEE International Conference on Computer Design, ICCD 2016
CountryUnited States
CityScottsdale
Period10/2/1610/5/16

Fingerprint

Neural networks
Hardware
Feature extraction
Field programmable gate arrays (FPGA)
Servers
Energy utilization
Processing

Keywords

  • Deep Convolutional Neural Networks
  • Deep Learning
  • Hardware-oriented Co-optimization
  • Stochastic Computing

ASJC Scopus subject areas

  • Hardware and Architecture

Cite this

Li, Z., Ren, A., Li, J., Qiu, Q., Wang, Y., & Yuan, B. (2016). DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks. In Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016 (pp. 678-681). [7753357] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCD.2016.7753357

DSCNN : Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks. / Li, Zhe; Ren, Ao; Li, Ji; Qiu, Qinru; Wang, Yanzhi; Yuan, Bo.

Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 678-681 7753357.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Li, Z, Ren, A, Li, J, Qiu, Q, Wang, Y & Yuan, B 2016, DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks. in Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016., 7753357, Institute of Electrical and Electronics Engineers Inc., pp. 678-681, 34th IEEE International Conference on Computer Design, ICCD 2016, Scottsdale, United States, 10/2/16. https://doi.org/10.1109/ICCD.2016.7753357
Li Z, Ren A, Li J, Qiu Q, Wang Y, Yuan B. DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks. In Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 678-681. 7753357 https://doi.org/10.1109/ICCD.2016.7753357
Li, Zhe ; Ren, Ao ; Li, Ji ; Qiu, Qinru ; Wang, Yanzhi ; Yuan, Bo. / DSCNN : Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks. Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016. Institute of Electrical and Electronics Engineers Inc., 2016. pp. 678-681
@inproceedings{f5a42f39237f4cc4974220eb96d15e46,
title = "DSCNN: Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks",
abstract = "Deep Convolutional Neural Networks (DCNN), a branch of Deep Neural Networks which use the deep graph with multiple processing layers, enables the convolutional model to finely abstract the high-level features behind an image. Large-scale applications using DCNN mainly operate in high-performance server clusters, GPUs or FPGA clusters; it is restricted to extend the applications onto mobile/wearable devices and Internet-of-Things (IoT) entities due to high power/energy consumption. Stochastic Computing is a promising method to overcome this shortcoming used in specific hardware-based systems. Many complex arithmetic operations can be implemented with very simple hardware logic in the SC framework, which alleviates the extensive computation complexity. The exploration of network-wise optimization and the revision of network structure with respect to stochastic computing based hardware design have not been discussed in previous work. In this paper, we investigate Deep Stochastic Convolutional Neural Network (DSCNN) for DCNN using stochastic computing. The essential calculation components using SC are designed and evaluated. We propose a joint optimization method to collaborate components guaranteeing a high calculation accuracy in each stage of the network. The structure of original DSCNN is revised to accommodate SC hardware design's simplicity. Experimental Results show that as opposed to software inspired feature extraction block in DSCNN, an optimized hardware oriented feature extraction block achieves as higher as 59.27{\%} calculation precision. And the optimized DSCNN can achieve only 3.48{\%} network test error rate compared to 27.83{\%} for baseline DSCNN using software inspired feature extraction block.",
keywords = "Deep Convolutional Neural Networks, Deep Learning, Hardware-oriented Co-optimization, Stochastic Computing",
author = "Zhe Li and Ao Ren and Ji Li and Qinru Qiu and Yanzhi Wang and Bo Yuan",
year = "2016",
month = "11",
day = "22",
doi = "10.1109/ICCD.2016.7753357",
language = "English (US)",
pages = "678--681",
booktitle = "Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - DSCNN

T2 - Hardware-oriented optimization for Stochastic Computing based Deep Convolutional Neural Networks

AU - Li, Zhe

AU - Ren, Ao

AU - Li, Ji

AU - Qiu, Qinru

AU - Wang, Yanzhi

AU - Yuan, Bo

PY - 2016/11/22

Y1 - 2016/11/22

N2 - Deep Convolutional Neural Networks (DCNN), a branch of Deep Neural Networks which use the deep graph with multiple processing layers, enables the convolutional model to finely abstract the high-level features behind an image. Large-scale applications using DCNN mainly operate in high-performance server clusters, GPUs or FPGA clusters; it is restricted to extend the applications onto mobile/wearable devices and Internet-of-Things (IoT) entities due to high power/energy consumption. Stochastic Computing is a promising method to overcome this shortcoming used in specific hardware-based systems. Many complex arithmetic operations can be implemented with very simple hardware logic in the SC framework, which alleviates the extensive computation complexity. The exploration of network-wise optimization and the revision of network structure with respect to stochastic computing based hardware design have not been discussed in previous work. In this paper, we investigate Deep Stochastic Convolutional Neural Network (DSCNN) for DCNN using stochastic computing. The essential calculation components using SC are designed and evaluated. We propose a joint optimization method to collaborate components guaranteeing a high calculation accuracy in each stage of the network. The structure of original DSCNN is revised to accommodate SC hardware design's simplicity. Experimental Results show that as opposed to software inspired feature extraction block in DSCNN, an optimized hardware oriented feature extraction block achieves as higher as 59.27% calculation precision. And the optimized DSCNN can achieve only 3.48% network test error rate compared to 27.83% for baseline DSCNN using software inspired feature extraction block.

AB - Deep Convolutional Neural Networks (DCNN), a branch of Deep Neural Networks which use the deep graph with multiple processing layers, enables the convolutional model to finely abstract the high-level features behind an image. Large-scale applications using DCNN mainly operate in high-performance server clusters, GPUs or FPGA clusters; it is restricted to extend the applications onto mobile/wearable devices and Internet-of-Things (IoT) entities due to high power/energy consumption. Stochastic Computing is a promising method to overcome this shortcoming used in specific hardware-based systems. Many complex arithmetic operations can be implemented with very simple hardware logic in the SC framework, which alleviates the extensive computation complexity. The exploration of network-wise optimization and the revision of network structure with respect to stochastic computing based hardware design have not been discussed in previous work. In this paper, we investigate Deep Stochastic Convolutional Neural Network (DSCNN) for DCNN using stochastic computing. The essential calculation components using SC are designed and evaluated. We propose a joint optimization method to collaborate components guaranteeing a high calculation accuracy in each stage of the network. The structure of original DSCNN is revised to accommodate SC hardware design's simplicity. Experimental Results show that as opposed to software inspired feature extraction block in DSCNN, an optimized hardware oriented feature extraction block achieves as higher as 59.27% calculation precision. And the optimized DSCNN can achieve only 3.48% network test error rate compared to 27.83% for baseline DSCNN using software inspired feature extraction block.

KW - Deep Convolutional Neural Networks

KW - Deep Learning

KW - Hardware-oriented Co-optimization

KW - Stochastic Computing

UR - http://www.scopus.com/inward/record.url?scp=85006716849&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85006716849&partnerID=8YFLogxK

U2 - 10.1109/ICCD.2016.7753357

DO - 10.1109/ICCD.2016.7753357

M3 - Conference contribution

AN - SCOPUS:85006716849

SP - 678

EP - 681

BT - Proceedings of the 34th IEEE International Conference on Computer Design, ICCD 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -