FPGA acceleration of recurrent neural network based language model

Sicheng Li, Chunpeng Wu, Hai Li, Boxun Li, Yu Wang, Qinru Qiu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

43 Citations (Scopus)

Abstract

Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in capturing semantics of sentences. However, the use of RNNLM has been greatly hindered for the high computation cost in training. This work presents an FPGA implementation framework for RNNLM training acceleration. At architectural level, we improve the parallelism of RNN training scheme and reduce the computing resource requirement for computation efficiency enhancement. The hardware implementation primarily targets at reducing data communication load. A multi-thread based computation engine is utilized which can successfully mask the long memory latency and reuse frequent accessed data. The evaluation based on the Microsoft Research Sentence Completion Challenge shows that the proposed FPGA implementation outperforms traditional class-based modest-size recurrent networks and obtains 46.2% in training accuracy. Moreover, experiments at different network sizes demonstrate a great scalability of the proposed framework.

Original languageEnglish (US)
Title of host publicationProceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages111-118
Number of pages8
ISBN (Print)9781479999699
DOIs
StatePublished - Jul 15 2015
Event23rd IEEE Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015 - Vancouver, Canada
Duration: May 3 2015May 5 2015

Other

Other23rd IEEE Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015
CountryCanada
CityVancouver
Period5/3/155/5/15

Fingerprint

Recurrent neural networks
Field programmable gate arrays (FPGA)
Scalability
Masks
Semantics
Engines
Hardware
Data storage equipment
Communication
Processing
Costs
Experiments

Keywords

  • Acceleration
  • FPGA
  • Language model
  • Recurrent neural network (RNN)

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Hardware and Architecture

Cite this

Li, S., Wu, C., Li, H., Li, B., Wang, Y., & Qiu, Q. (2015). FPGA acceleration of recurrent neural network based language model. In Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015 (pp. 111-118). [7160054] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FCCM.2015.50

FPGA acceleration of recurrent neural network based language model. / Li, Sicheng; Wu, Chunpeng; Li, Hai; Li, Boxun; Wang, Yu; Qiu, Qinru.

Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015. Institute of Electrical and Electronics Engineers Inc., 2015. p. 111-118 7160054.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Li, S, Wu, C, Li, H, Li, B, Wang, Y & Qiu, Q 2015, FPGA acceleration of recurrent neural network based language model. in Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015., 7160054, Institute of Electrical and Electronics Engineers Inc., pp. 111-118, 23rd IEEE Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015, Vancouver, Canada, 5/3/15. https://doi.org/10.1109/FCCM.2015.50
Li S, Wu C, Li H, Li B, Wang Y, Qiu Q. FPGA acceleration of recurrent neural network based language model. In Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 111-118. 7160054 https://doi.org/10.1109/FCCM.2015.50
Li, Sicheng ; Wu, Chunpeng ; Li, Hai ; Li, Boxun ; Wang, Yu ; Qiu, Qinru. / FPGA acceleration of recurrent neural network based language model. Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 111-118
@inproceedings{20db51487f954a5f99aaaac5a4b8cf29,
title = "FPGA acceleration of recurrent neural network based language model",
abstract = "Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in capturing semantics of sentences. However, the use of RNNLM has been greatly hindered for the high computation cost in training. This work presents an FPGA implementation framework for RNNLM training acceleration. At architectural level, we improve the parallelism of RNN training scheme and reduce the computing resource requirement for computation efficiency enhancement. The hardware implementation primarily targets at reducing data communication load. A multi-thread based computation engine is utilized which can successfully mask the long memory latency and reuse frequent accessed data. The evaluation based on the Microsoft Research Sentence Completion Challenge shows that the proposed FPGA implementation outperforms traditional class-based modest-size recurrent networks and obtains 46.2{\%} in training accuracy. Moreover, experiments at different network sizes demonstrate a great scalability of the proposed framework.",
keywords = "Acceleration, FPGA, Language model, Recurrent neural network (RNN)",
author = "Sicheng Li and Chunpeng Wu and Hai Li and Boxun Li and Yu Wang and Qinru Qiu",
year = "2015",
month = "7",
day = "15",
doi = "10.1109/FCCM.2015.50",
language = "English (US)",
isbn = "9781479999699",
pages = "111--118",
booktitle = "Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - FPGA acceleration of recurrent neural network based language model

AU - Li, Sicheng

AU - Wu, Chunpeng

AU - Li, Hai

AU - Li, Boxun

AU - Wang, Yu

AU - Qiu, Qinru

PY - 2015/7/15

Y1 - 2015/7/15

N2 - Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in capturing semantics of sentences. However, the use of RNNLM has been greatly hindered for the high computation cost in training. This work presents an FPGA implementation framework for RNNLM training acceleration. At architectural level, we improve the parallelism of RNN training scheme and reduce the computing resource requirement for computation efficiency enhancement. The hardware implementation primarily targets at reducing data communication load. A multi-thread based computation engine is utilized which can successfully mask the long memory latency and reuse frequent accessed data. The evaluation based on the Microsoft Research Sentence Completion Challenge shows that the proposed FPGA implementation outperforms traditional class-based modest-size recurrent networks and obtains 46.2% in training accuracy. Moreover, experiments at different network sizes demonstrate a great scalability of the proposed framework.

AB - Recurrent neural network (RNN) based language model (RNNLM) is a biologically inspired model for natural language processing. It records the historical information through additional recurrent connections and therefore is very effective in capturing semantics of sentences. However, the use of RNNLM has been greatly hindered for the high computation cost in training. This work presents an FPGA implementation framework for RNNLM training acceleration. At architectural level, we improve the parallelism of RNN training scheme and reduce the computing resource requirement for computation efficiency enhancement. The hardware implementation primarily targets at reducing data communication load. A multi-thread based computation engine is utilized which can successfully mask the long memory latency and reuse frequent accessed data. The evaluation based on the Microsoft Research Sentence Completion Challenge shows that the proposed FPGA implementation outperforms traditional class-based modest-size recurrent networks and obtains 46.2% in training accuracy. Moreover, experiments at different network sizes demonstrate a great scalability of the proposed framework.

KW - Acceleration

KW - FPGA

KW - Language model

KW - Recurrent neural network (RNN)

UR - http://www.scopus.com/inward/record.url?scp=84943378851&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84943378851&partnerID=8YFLogxK

U2 - 10.1109/FCCM.2015.50

DO - 10.1109/FCCM.2015.50

M3 - Conference contribution

AN - SCOPUS:84943378851

SN - 9781479999699

SP - 111

EP - 118

BT - Proceedings - 2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines, FCCM 2015

PB - Institute of Electrical and Electronics Engineers Inc.

ER -