A Deep Dive into the Google Cluster Workload Traces: Analyzing the Application Failure Characteristics and User Behaviors

Faisal Haque Bappy, Tariqul Islam, Tarannum Shaila Zaman, Raiful Hasan, Carlos Caicedo

Research output: Chapter in Book/Entry/PoemConference contribution

Abstract

Large-scale cloud data centers have gained popularity due to their high availability, rapid elasticity, scalability, and low cost. However, current data centers continue to have high failure rates due to the lack of proper resource utilization and early failure detection. To maximize resource efficiency and reduce failure rates in large-scale cloud data centers, it is crucial to understand the workload and failure characteristics. In this paper, we perform a deep analysis of the 2019 Google Cluster Trace Dataset, which contains 2.4TiB of workload traces from eight different clusters around the world. We explore the characteristics of failed and killed jobs in Google's production cloud and attempt to correlate them with key attributes such as resource usage, job priority, scheduling class, job duration, and the number of task resubmissions. Our analysis reveals several important characteristics of failed jobs that contribute to job failure and hence, could be used for developing an early failure prediction system. Also, we present a novel usage analysis to identify heterogeneity in jobs and tasks submitted by users. We are able to identify specific users who control more than half of all collection events on a single cluster. We contend that these characteristics could be useful in developing an early job failure prediction system that could be utilized for dynamic rescheduling of the job scheduler and thus improving resource utilization in large-scale cloud data centers while reducing failure rates.

Original languageEnglish (US)
Title of host publicationProceedings - 2023 International Conference on Future Internet of Things and Cloud, FiCloud 2023
EditorsIrfan Awan, Muhammad Younas, Markus Aleksy
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages103-108
Number of pages6
ISBN (Electronic)9798350316353
DOIs
StatePublished - 2023
Event10th International Conference on Future Internet of Things and Cloud, FiCloud 2023 - Hybrid, Marrakech, Morocco
Duration: Aug 14 2023Aug 16 2023

Publication series

NameProceedings - 2023 International Conference on Future Internet of Things and Cloud, FiCloud 2023

Conference

Conference10th International Conference on Future Internet of Things and Cloud, FiCloud 2023
Country/TerritoryMorocco
CityHybrid, Marrakech
Period8/14/238/16/23

Keywords

  • Cloud Availability
  • Cloud Reliability
  • Failure Characterization
  • Fault Tolerance
  • Google Cluster Trace

ASJC Scopus subject areas

  • Computer Networks and Communications
  • Computer Science Applications
  • Hardware and Architecture
  • Information Systems
  • Information Systems and Management

Fingerprint

Dive into the research topics of 'A Deep Dive into the Google Cluster Workload Traces: Analyzing the Application Failure Characteristics and User Behaviors'. Together they form a unique fingerprint.

Cite this