Achieving Agile Big Data Science: The Evolution of a Team's Agile Process Methodology

Jeffrey S. Saltz, Ivan Shamshurin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

While there has been a rapid increase in the use of data science and the related field of big data, there has been minimal discussion on how teams using these techniques should best plan, coordinate and communicate their activities. To help address this gap, this paper reports on a mixed method qualitative study exploring how a big data science team within a Fortune 500 organization used two different agile process methodologies. The study helps clarify the concept of agility within a big data science project, as well as the key process challenges teams encounter when executing a big data science project. Specifically, three key issues were identified: (a) the challenge in task duration estimation, (b) how to account for team members that might be pulled onto other tasks for short bursts and (c) coordination challenges across the different groups within the big data science team. Our findings help explain how different process methodologies might mitigate or exacerbate these challenges and supports previous research showing that big data science teams would benefit from an increased focus on their process methodology and that adopting an Agile Kanban methodology, which focuses on minimizing work-in-progress, could prove beneficial for many big data science teams.

Original languageEnglish (US)
Title of host publicationProceedings - 2019 IEEE International Conference on Big Data, Big Data 2019
EditorsChaitanya Baru, Jun Huan, Latifur Khan, Xiaohua Tony Hu, Ronay Ak, Yuanyuan Tian, Roger Barga, Carlo Zaniolo, Kisung Lee, Yanfang Fanny Ye
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages3477-3485
Number of pages9
ISBN (Electronic)9781728108582
DOIs
StatePublished - Dec 2019
Externally publishedYes
Event2019 IEEE International Conference on Big Data, Big Data 2019 - Los Angeles, United States
Duration: Dec 9 2019Dec 12 2019

Publication series

NameProceedings - 2019 IEEE International Conference on Big Data, Big Data 2019

Conference

Conference2019 IEEE International Conference on Big Data, Big Data 2019
CountryUnited States
CityLos Angeles
Period12/9/1912/12/19

Keywords

  • Agile
  • Big Data Science
  • Process Methodology

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Information Systems
  • Information Systems and Management

Fingerprint Dive into the research topics of 'Achieving Agile Big Data Science: The Evolution of a Team's Agile Process Methodology'. Together they form a unique fingerprint.

  • Cite this

    Saltz, J. S., & Shamshurin, I. (2019). Achieving Agile Big Data Science: The Evolution of a Team's Agile Process Methodology. In C. Baru, J. Huan, L. Khan, X. T. Hu, R. Ak, Y. Tian, R. Barga, C. Zaniolo, K. Lee, & Y. F. Ye (Eds.), Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019 (pp. 3477-3485). [9005493] (Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData47090.2019.9005493