Teaching citizen scientists to categorize glitches using machine learning guided training

Corey Jackson, Carsten Østerlund, Kevin Crowston, Mahboobeh Harandi, Sarah Allen, Sara Bahaadini, Scotty Coughlin, Vicky Kalogera, Aggelos Katsaggelos, Shane Larson, Neda Rohani, Joshua Smith, Laura Trouille, Michael Zevin

Research output: Contribution to journalArticle

Abstract

Existing literature points to scaffolded training as an effective yet resource-intensive approach to help newcomers learn and stay motivated. Experts need to select relevant learning materials and continuously assess learners' progress. Peer production communities such as Wikipedia and Open Source Software Development projects face the additional problem of turning volunteers into productive participants as soon as possible. To address these challenges, we designed and tested a training regime combining scaffolded instruction and machine learning to select learning materials and gradually introduces new materials to individuals as their competences improve. We evaluated the training regime on 386 participants that contribute to Gravity Spy, an online citizen science project where people are asked to categorize glitches to assist scientists in the search for gravitational waves. Volunteers were assigned to one of two conditions; (1) a machine learning guided training (MLGT) system that continuously assesses volunteers skill level and adjusts the learning materials or (2) an unscaffolded training program where all learning materials were administered at once. Our analysis revealed that volunteers in the MLGT condition were more accurate on the categorization task (an average accuracy of 90% vs. 54%), executed more tasks (an average of 228 vs. 121 tasks), and were retained for a longer period (an average of 2.5 vs. 2 sessions) than volunteers in the unscaffolded training. The results suggest that MLGT is an effective pedagogical approach for training volunteers in categorization tasks and increases volunteers’ motivation.

Original languageEnglish (US)
Article number106198
JournalComputers in Human Behavior
Volume105
DOIs
StatePublished - Apr 2020

Fingerprint

Learning systems
Volunteers
Teaching
Learning
Gravitation
Machine Learning
Mental Competency
Gravity waves
Motivation
Software
Software engineering
Education

Keywords

  • Citizen science
  • Experiment
  • Learning
  • Online communities
  • Scaffolding
  • Training
  • User studies
  • Zooniverse

ASJC Scopus subject areas

  • Arts and Humanities (miscellaneous)
  • Human-Computer Interaction
  • Psychology(all)

Cite this

Teaching citizen scientists to categorize glitches using machine learning guided training. / Jackson, Corey; Østerlund, Carsten; Crowston, Kevin; Harandi, Mahboobeh; Allen, Sarah; Bahaadini, Sara; Coughlin, Scotty; Kalogera, Vicky; Katsaggelos, Aggelos; Larson, Shane; Rohani, Neda; Smith, Joshua; Trouille, Laura; Zevin, Michael.

In: Computers in Human Behavior, Vol. 105, 106198, 04.2020.

Research output: Contribution to journalArticle

Jackson, C, Østerlund, C, Crowston, K, Harandi, M, Allen, S, Bahaadini, S, Coughlin, S, Kalogera, V, Katsaggelos, A, Larson, S, Rohani, N, Smith, J, Trouille, L & Zevin, M 2020, 'Teaching citizen scientists to categorize glitches using machine learning guided training', Computers in Human Behavior, vol. 105, 106198. https://doi.org/10.1016/j.chb.2019.106198
Jackson, Corey ; Østerlund, Carsten ; Crowston, Kevin ; Harandi, Mahboobeh ; Allen, Sarah ; Bahaadini, Sara ; Coughlin, Scotty ; Kalogera, Vicky ; Katsaggelos, Aggelos ; Larson, Shane ; Rohani, Neda ; Smith, Joshua ; Trouille, Laura ; Zevin, Michael. / Teaching citizen scientists to categorize glitches using machine learning guided training. In: Computers in Human Behavior. 2020 ; Vol. 105.
@article{dcfd89ffeda34439b80b7b851017818c,
title = "Teaching citizen scientists to categorize glitches using machine learning guided training",
abstract = "Existing literature points to scaffolded training as an effective yet resource-intensive approach to help newcomers learn and stay motivated. Experts need to select relevant learning materials and continuously assess learners' progress. Peer production communities such as Wikipedia and Open Source Software Development projects face the additional problem of turning volunteers into productive participants as soon as possible. To address these challenges, we designed and tested a training regime combining scaffolded instruction and machine learning to select learning materials and gradually introduces new materials to individuals as their competences improve. We evaluated the training regime on 386 participants that contribute to Gravity Spy, an online citizen science project where people are asked to categorize glitches to assist scientists in the search for gravitational waves. Volunteers were assigned to one of two conditions; (1) a machine learning guided training (MLGT) system that continuously assesses volunteers skill level and adjusts the learning materials or (2) an unscaffolded training program where all learning materials were administered at once. Our analysis revealed that volunteers in the MLGT condition were more accurate on the categorization task (an average accuracy of 90{\%} vs. 54{\%}), executed more tasks (an average of 228 vs. 121 tasks), and were retained for a longer period (an average of 2.5 vs. 2 sessions) than volunteers in the unscaffolded training. The results suggest that MLGT is an effective pedagogical approach for training volunteers in categorization tasks and increases volunteers’ motivation.",
keywords = "Citizen science, Experiment, Learning, Online communities, Scaffolding, Training, User studies, Zooniverse",
author = "Corey Jackson and Carsten {\O}sterlund and Kevin Crowston and Mahboobeh Harandi and Sarah Allen and Sara Bahaadini and Scotty Coughlin and Vicky Kalogera and Aggelos Katsaggelos and Shane Larson and Neda Rohani and Joshua Smith and Laura Trouille and Michael Zevin",
year = "2020",
month = "4",
doi = "10.1016/j.chb.2019.106198",
language = "English (US)",
volume = "105",
journal = "Computers in Human Behavior",
issn = "0747-5632",
publisher = "Elsevier",

}

TY - JOUR

T1 - Teaching citizen scientists to categorize glitches using machine learning guided training

AU - Jackson, Corey

AU - Østerlund, Carsten

AU - Crowston, Kevin

AU - Harandi, Mahboobeh

AU - Allen, Sarah

AU - Bahaadini, Sara

AU - Coughlin, Scotty

AU - Kalogera, Vicky

AU - Katsaggelos, Aggelos

AU - Larson, Shane

AU - Rohani, Neda

AU - Smith, Joshua

AU - Trouille, Laura

AU - Zevin, Michael

PY - 2020/4

Y1 - 2020/4

N2 - Existing literature points to scaffolded training as an effective yet resource-intensive approach to help newcomers learn and stay motivated. Experts need to select relevant learning materials and continuously assess learners' progress. Peer production communities such as Wikipedia and Open Source Software Development projects face the additional problem of turning volunteers into productive participants as soon as possible. To address these challenges, we designed and tested a training regime combining scaffolded instruction and machine learning to select learning materials and gradually introduces new materials to individuals as their competences improve. We evaluated the training regime on 386 participants that contribute to Gravity Spy, an online citizen science project where people are asked to categorize glitches to assist scientists in the search for gravitational waves. Volunteers were assigned to one of two conditions; (1) a machine learning guided training (MLGT) system that continuously assesses volunteers skill level and adjusts the learning materials or (2) an unscaffolded training program where all learning materials were administered at once. Our analysis revealed that volunteers in the MLGT condition were more accurate on the categorization task (an average accuracy of 90% vs. 54%), executed more tasks (an average of 228 vs. 121 tasks), and were retained for a longer period (an average of 2.5 vs. 2 sessions) than volunteers in the unscaffolded training. The results suggest that MLGT is an effective pedagogical approach for training volunteers in categorization tasks and increases volunteers’ motivation.

AB - Existing literature points to scaffolded training as an effective yet resource-intensive approach to help newcomers learn and stay motivated. Experts need to select relevant learning materials and continuously assess learners' progress. Peer production communities such as Wikipedia and Open Source Software Development projects face the additional problem of turning volunteers into productive participants as soon as possible. To address these challenges, we designed and tested a training regime combining scaffolded instruction and machine learning to select learning materials and gradually introduces new materials to individuals as their competences improve. We evaluated the training regime on 386 participants that contribute to Gravity Spy, an online citizen science project where people are asked to categorize glitches to assist scientists in the search for gravitational waves. Volunteers were assigned to one of two conditions; (1) a machine learning guided training (MLGT) system that continuously assesses volunteers skill level and adjusts the learning materials or (2) an unscaffolded training program where all learning materials were administered at once. Our analysis revealed that volunteers in the MLGT condition were more accurate on the categorization task (an average accuracy of 90% vs. 54%), executed more tasks (an average of 228 vs. 121 tasks), and were retained for a longer period (an average of 2.5 vs. 2 sessions) than volunteers in the unscaffolded training. The results suggest that MLGT is an effective pedagogical approach for training volunteers in categorization tasks and increases volunteers’ motivation.

KW - Citizen science

KW - Experiment

KW - Learning

KW - Online communities

KW - Scaffolding

KW - Training

KW - User studies

KW - Zooniverse

UR - http://www.scopus.com/inward/record.url?scp=85075531612&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85075531612&partnerID=8YFLogxK

U2 - 10.1016/j.chb.2019.106198

DO - 10.1016/j.chb.2019.106198

M3 - Article

AN - SCOPUS:85075531612

VL - 105

JO - Computers in Human Behavior

JF - Computers in Human Behavior

SN - 0747-5632

M1 - 106198

ER -