EXTRA: An Experience-driven Control Framework for Distributed Stream Data Processing with a Variable Number of Threads

Teng Li, Zhiyuan Xu, Jian Tang, Kun Wu, Yanzhi Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we present design, implementation and evaluation of a control framework, EXTRA (Experience-driven conTRol frAmework), for scheduling in general-purpose Distributed Stream Data Processing Systems (DSDPSs). Our design is novel due to the following reasons. First, EXTRA enables a DSDPS to dynamically change the number of threads on the fly according to system states and demands. Most existing methods, however, use a fixed number of threads to carry workload (for each processing unit of an application), which is specified by a user in advance and does not change during runtime. So our design introduces a whole new dimension for control in DSDPSs, which has a great potential to significantly improve system flexibility and efficiency, but makes the scheduling problem much harder. Second, EXTRA leverages an experience/data driven model-free approach for dynamic control using the emerging Deep Reinforcement Learning (DRL), which enables a DSDPS to learn the best way to control itself from its own experience just as a human learns a skill (such as driving and swimming) without any accurate and mathematically solvable model. We implemented it based on a widely-used DSDPS, Apache Storm, and evaluated its performance with three representative Stream Data Processing (SDP) applications: continuous queries, word count (stream version) and log stream processing. Particularly, we performed experiments under realistic settings (where multiple application instances are mixed up together), rather than a simplified setting (where experiments are conducted only on a single application instance) used in most related works. Extensive experimental results show: 1) Compared to Storm's default scheduler and the state-of-the-art model-based method, EXTRA substantially reduces average end-to-end tuple processing time by 39.6% and 21.6% respectively on average. 2) EXTRA does lead to more flexible and efficient stream data processing by enabling the use of a variable number of threads. 3) EXTRA is robust in a highly dynamic environment with significant workload change.

Original languageEnglish (US)
Title of host publication2021 IEEE/ACM 29th International Symposium on Quality of Service, IWQOS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9781665414944
DOIs
StatePublished - Jun 25 2021
Event29th IEEE/ACM International Symposium on Quality of Service, IWQOS 2021 - Virtual, Tokyo, Japan
Duration: Jun 25 2021Jun 28 2021

Publication series

Name2021 IEEE/ACM 29th International Symposium on Quality of Service, IWQOS 2021

Conference

Conference29th IEEE/ACM International Symposium on Quality of Service, IWQOS 2021
Country/TerritoryJapan
CityVirtual, Tokyo
Period6/25/216/28/21

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Information Systems and Management
  • Safety, Risk, Reliability and Quality

Fingerprint

Dive into the research topics of 'EXTRA: An Experience-driven Control Framework for Distributed Stream Data Processing with a Variable Number of Threads'. Together they form a unique fingerprint.

Cite this