Reinforcement learning with analogue memristor arrays

Zhongrui Wang, Can Li, Wenhao Song, Mingyi Rao, Daniel Belkin, Yunning Li, Peng Yan, Hao Jiang, Peng Lin, Miao Hu, John Paul Strachan, Ning Ge, Mark Barnell, Qing Wu, Andrew G. Barto, Qinru Qiu, R. Stanley Williams, Qiangfei Xia, J. Joshua Yang

Research output: Contribution to journalArticle

19 Citations (Scopus)

Abstract

Reinforcement learning algorithms that use deep neural networks are a promising approach for the development of machines that can acquire knowledge and solve problems without human input or supervision. At present, however, these algorithms are implemented in software running on relatively standard complementary metal–oxide–semiconductor digital platforms, where performance will be constrained by the limits of Moore’s law and von Neumann architecture. Here, we report an experimental demonstration of reinforcement learning on a three-layer 1-transistor 1-memristor (1T1R) network using a modified learning algorithm tailored for our hybrid analogue–digital platform. To illustrate the capabilities of our approach in robust in situ training without the need for a model, we performed two classic control problems: the cart–pole and mountain car simulations. We also show that, compared with conventional digital systems in real-world reinforcement learning tasks, our hybrid analogue–digital computing system has the potential to achieve a significant boost in speed and energy efficiency.

Original languageEnglish (US)
Pages (from-to)115-124
Number of pages10
JournalNature Electronics
Volume2
Issue number3
DOIs
StatePublished - Mar 1 2019

Fingerprint

Memristors
Reinforcement learning
reinforcement
learning
analogs
Learning algorithms
platforms
digital systems
Energy efficiency
Transistors
Railroad cars
Demonstrations
acceleration (physics)
mountains
education
transistors
computer programs
simulation
energy

ASJC Scopus subject areas

  • Electrical and Electronic Engineering
  • Electronic, Optical and Magnetic Materials
  • Instrumentation

Cite this

Wang, Z., Li, C., Song, W., Rao, M., Belkin, D., Li, Y., ... Yang, J. J. (2019). Reinforcement learning with analogue memristor arrays. Nature Electronics, 2(3), 115-124. https://doi.org/10.1038/s41928-019-0221-6

Reinforcement learning with analogue memristor arrays. / Wang, Zhongrui; Li, Can; Song, Wenhao; Rao, Mingyi; Belkin, Daniel; Li, Yunning; Yan, Peng; Jiang, Hao; Lin, Peng; Hu, Miao; Strachan, John Paul; Ge, Ning; Barnell, Mark; Wu, Qing; Barto, Andrew G.; Qiu, Qinru; Williams, R. Stanley; Xia, Qiangfei; Yang, J. Joshua.

In: Nature Electronics, Vol. 2, No. 3, 01.03.2019, p. 115-124.

Research output: Contribution to journalArticle

Wang, Z, Li, C, Song, W, Rao, M, Belkin, D, Li, Y, Yan, P, Jiang, H, Lin, P, Hu, M, Strachan, JP, Ge, N, Barnell, M, Wu, Q, Barto, AG, Qiu, Q, Williams, RS, Xia, Q & Yang, JJ 2019, 'Reinforcement learning with analogue memristor arrays', Nature Electronics, vol. 2, no. 3, pp. 115-124. https://doi.org/10.1038/s41928-019-0221-6
Wang Z, Li C, Song W, Rao M, Belkin D, Li Y et al. Reinforcement learning with analogue memristor arrays. Nature Electronics. 2019 Mar 1;2(3):115-124. https://doi.org/10.1038/s41928-019-0221-6
Wang, Zhongrui ; Li, Can ; Song, Wenhao ; Rao, Mingyi ; Belkin, Daniel ; Li, Yunning ; Yan, Peng ; Jiang, Hao ; Lin, Peng ; Hu, Miao ; Strachan, John Paul ; Ge, Ning ; Barnell, Mark ; Wu, Qing ; Barto, Andrew G. ; Qiu, Qinru ; Williams, R. Stanley ; Xia, Qiangfei ; Yang, J. Joshua. / Reinforcement learning with analogue memristor arrays. In: Nature Electronics. 2019 ; Vol. 2, No. 3. pp. 115-124.
@article{bb494337cb7b4fd59352810158c96684,
title = "Reinforcement learning with analogue memristor arrays",
abstract = "Reinforcement learning algorithms that use deep neural networks are a promising approach for the development of machines that can acquire knowledge and solve problems without human input or supervision. At present, however, these algorithms are implemented in software running on relatively standard complementary metal–oxide–semiconductor digital platforms, where performance will be constrained by the limits of Moore’s law and von Neumann architecture. Here, we report an experimental demonstration of reinforcement learning on a three-layer 1-transistor 1-memristor (1T1R) network using a modified learning algorithm tailored for our hybrid analogue–digital platform. To illustrate the capabilities of our approach in robust in situ training without the need for a model, we performed two classic control problems: the cart–pole and mountain car simulations. We also show that, compared with conventional digital systems in real-world reinforcement learning tasks, our hybrid analogue–digital computing system has the potential to achieve a significant boost in speed and energy efficiency.",
author = "Zhongrui Wang and Can Li and Wenhao Song and Mingyi Rao and Daniel Belkin and Yunning Li and Peng Yan and Hao Jiang and Peng Lin and Miao Hu and Strachan, {John Paul} and Ning Ge and Mark Barnell and Qing Wu and Barto, {Andrew G.} and Qinru Qiu and Williams, {R. Stanley} and Qiangfei Xia and Yang, {J. Joshua}",
year = "2019",
month = "3",
day = "1",
doi = "10.1038/s41928-019-0221-6",
language = "English (US)",
volume = "2",
pages = "115--124",
journal = "Nature Electronics",
issn = "2520-1131",
publisher = "Nature Publishing Group",
number = "3",

}

TY - JOUR

T1 - Reinforcement learning with analogue memristor arrays

AU - Wang, Zhongrui

AU - Li, Can

AU - Song, Wenhao

AU - Rao, Mingyi

AU - Belkin, Daniel

AU - Li, Yunning

AU - Yan, Peng

AU - Jiang, Hao

AU - Lin, Peng

AU - Hu, Miao

AU - Strachan, John Paul

AU - Ge, Ning

AU - Barnell, Mark

AU - Wu, Qing

AU - Barto, Andrew G.

AU - Qiu, Qinru

AU - Williams, R. Stanley

AU - Xia, Qiangfei

AU - Yang, J. Joshua

PY - 2019/3/1

Y1 - 2019/3/1

N2 - Reinforcement learning algorithms that use deep neural networks are a promising approach for the development of machines that can acquire knowledge and solve problems without human input or supervision. At present, however, these algorithms are implemented in software running on relatively standard complementary metal–oxide–semiconductor digital platforms, where performance will be constrained by the limits of Moore’s law and von Neumann architecture. Here, we report an experimental demonstration of reinforcement learning on a three-layer 1-transistor 1-memristor (1T1R) network using a modified learning algorithm tailored for our hybrid analogue–digital platform. To illustrate the capabilities of our approach in robust in situ training without the need for a model, we performed two classic control problems: the cart–pole and mountain car simulations. We also show that, compared with conventional digital systems in real-world reinforcement learning tasks, our hybrid analogue–digital computing system has the potential to achieve a significant boost in speed and energy efficiency.

AB - Reinforcement learning algorithms that use deep neural networks are a promising approach for the development of machines that can acquire knowledge and solve problems without human input or supervision. At present, however, these algorithms are implemented in software running on relatively standard complementary metal–oxide–semiconductor digital platforms, where performance will be constrained by the limits of Moore’s law and von Neumann architecture. Here, we report an experimental demonstration of reinforcement learning on a three-layer 1-transistor 1-memristor (1T1R) network using a modified learning algorithm tailored for our hybrid analogue–digital platform. To illustrate the capabilities of our approach in robust in situ training without the need for a model, we performed two classic control problems: the cart–pole and mountain car simulations. We also show that, compared with conventional digital systems in real-world reinforcement learning tasks, our hybrid analogue–digital computing system has the potential to achieve a significant boost in speed and energy efficiency.

UR - http://www.scopus.com/inward/record.url?scp=85063037929&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85063037929&partnerID=8YFLogxK

U2 - 10.1038/s41928-019-0221-6

DO - 10.1038/s41928-019-0221-6

M3 - Article

AN - SCOPUS:85063037929

VL - 2

SP - 115

EP - 124

JO - Nature Electronics

JF - Nature Electronics

SN - 2520-1131

IS - 3

ER -