Asynchronous multitask reinforcement learning with dropout for continuous control

Zilong Jiao, Jae Oh

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Scopus citations

Abstract

Deep reinforcement learning is sample inefficient for solving complex tasks. Recently, multitask reinforcement learning has received increased attention because of its ability to learn general policies with improved sample efficiency. In multitask reinforcement learning, a single agent must learn multiple related tasks, either sequentially or simultaneously. Based on the DDPG algorithm, this paper presents Asyn-DDPG, which asynchronously learns a multitask policy for continuous control with simultaneous worker agents. We empirically found that sparse policy gradients can significantly reduce interference among conflicting tasks and make multitask learning more stable and sample efficient. To ensure the sparsity of gradients evaluated for each task, Asyn-DDPG represents both actor and critic functions as deep neural networks and regularizes them using Dropout. During training, worker agents share the actor and the critic functions, and asynchronously optimize them using task-specific gradients. For evaluating Asyn-DDPG, we proposed robotic navigation tasks based on realistically simulated robots and physics-enabled maze-like environments. Although the number of tasks used in our experiment is small, each task is conducted based on a real-world setting and posts a challenging environment. Through extensive evaluation, we demonstrate that Dropout regularization can effectively stabilize asynchronous learning and enable Asyn-DDPG to outperform DDPG significantly. Also, Asyn-DDPG was able to learn a multitask policy that can be well generalized for handling environments unseen during training.

Original languageEnglish (US)
Title of host publicationProceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
EditorsM. Arif Wani, Taghi M. Khoshgoftaar, Dingding Wang, Huanjing Wang, Naeem Seliya
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages529-534
Number of pages6
ISBN (Electronic)9781728145495
DOIs
StatePublished - Dec 2019
Event18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019 - Boca Raton, United States
Duration: Dec 16 2019Dec 19 2019

Publication series

NameProceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019

Conference

Conference18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019
CountryUnited States
CityBoca Raton
Period12/16/1912/19/19

Keywords

  • Asynchronous method
  • Continuous control
  • Deep reinforcement learning
  • Multitask reinforcement learning
  • Partial observability

ASJC Scopus subject areas

  • Strategy and Management
  • Artificial Intelligence
  • Computer Science Applications
  • Decision Sciences (miscellaneous)
  • Signal Processing
  • Media Technology

Fingerprint Dive into the research topics of 'Asynchronous multitask reinforcement learning with dropout for continuous control'. Together they form a unique fingerprint.

  • Cite this

    Jiao, Z., & Oh, J. (2019). Asynchronous multitask reinforcement learning with dropout for continuous control. In M. A. Wani, T. M. Khoshgoftaar, D. Wang, H. Wang, & N. Seliya (Eds.), Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019 (pp. 529-534). [8999228] (Proceedings - 18th IEEE International Conference on Machine Learning and Applications, ICMLA 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA.2019.00099