A real-time actor-critic architecture for continuous control

Zilong Jiao, Jae Oh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Reinforcement learning achieved impressive results in various challenging artificial environments and demonstrated its practical potential. In a real-world environment, an agent operates in continuous time, and it is unavoidable for the agent to have control delay. In the context of reinforcement learning, we define control delays as the time delay before an agent actuates an action in a particular state. The high-variance of control delay can destabilize an agent’s learning performance and make an environment Non-Markovian, creating a challenging situation for reinforcement learning algorithms. To address this issue, we present a scalable real-time architecture, RTAC, for reinforcement learning to continuous control. A reinforcement learning application usually consists of a policy training phase in simulation and the deployment phase of the learned policy to the real-world environment. We evaluated RTAC in a simulated environment close to its real-world setting, where agents operate in real-time and learn to map high-dimensional sensor data to continuous actions. In extensive experiments, RTAC was able to stabilize control delay and consistently learn optimal policies. Additionally, we demonstrated that RTAC was suitable for distributed learning even in the presence of control delay.

Original languageEnglish (US)
Title of host publicationTrends in Artificial Intelligence Theory and Applications. Artificial Intelligence Practices - 33rd International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2020, Proceedings
EditorsHamido Fujita, Jun Sasaki, Philippe Fournier-Viger, Moonis Ali
PublisherSpringer Science and Business Media Deutschland GmbH
Pages545-556
Number of pages12
ISBN (Print)9783030557881
DOIs
StatePublished - 2020
Event33rd International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2020 - Kitakyushu, Japan
Duration: Sep 22 2020Sep 25 2020

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume12144 LNAI
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference33rd International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2020
Country/TerritoryJapan
CityKitakyushu
Period9/22/209/25/20

Keywords

  • Actor-critic methods
  • Continuous control
  • Control delay
  • Distributed learning
  • Real-time architecture

ASJC Scopus subject areas

  • Theoretical Computer Science
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'A real-time actor-critic architecture for continuous control'. Together they form a unique fingerprint.

Cite this