Reliable crowdsourcing for multi-class labeling using coding theory

Aditya Vempaty, Lav R. Varshney, Pramod K. Varshney

Research output: Contribution to journalArticlepeer-review

40 Scopus citations

Abstract

Crowdsourcing systems often have crowd workers that perform unreliable work on the task they are assigned. In this paper, we propose the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification despite unreliable crowd workers. Coding theory based techniques also allow us to pose easy-to-answer binary questions to the crowd workers. We consider three different crowdsourcing models: systems with independent crowd workers, systems with peer-dependent reward schemes, and systems where workers have common sources of information. For each of these models, we analyze classification performance with the proposed coding-based scheme. We develop an ordering principle for the quality of crowds and describe how system performance changes with the quality of the crowd. We also show that pairing among workers and diversification of the questions help in improving system performance. We demonstrate the effectiveness of the proposed coding-based scheme using both simulated data and real datasets from Amazon Mechanical Turk, a crowdsourcing microtask platform. Results suggest that use of good codes may improve the performance of the crowdsourcing task over typical majority-voting approaches.

Original languageEnglish (US)
Article number6784318
Pages (from-to)667-679
Number of pages13
JournalIEEE Journal on Selected Topics in Signal Processing
Volume8
Issue number4
DOIs
StatePublished - Aug 2014

Keywords

  • Crowdsourcing
  • error-control codes
  • multi-class labeling
  • quality assurance

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Reliable crowdsourcing for multi-class labeling using coding theory'. Together they form a unique fingerprint.

Cite this