Reliable crowdsourcing for multi-class labeling using coding theory

Aditya Vempaty, Lav R. Varshney, Pramod K. Varshney

Research output: Contribution to journalArticlepeer-review

47 Scopus citations


Crowdsourcing systems often have crowd workers that perform unreliable work on the task they are assigned. In this paper, we propose the use of error-control codes and decoding algorithms to design crowdsourcing systems for reliable classification despite unreliable crowd workers. Coding theory based techniques also allow us to pose easy-to-answer binary questions to the crowd workers. We consider three different crowdsourcing models: systems with independent crowd workers, systems with peer-dependent reward schemes, and systems where workers have common sources of information. For each of these models, we analyze classification performance with the proposed coding-based scheme. We develop an ordering principle for the quality of crowds and describe how system performance changes with the quality of the crowd. We also show that pairing among workers and diversification of the questions help in improving system performance. We demonstrate the effectiveness of the proposed coding-based scheme using both simulated data and real datasets from Amazon Mechanical Turk, a crowdsourcing microtask platform. Results suggest that use of good codes may improve the performance of the crowdsourcing task over typical majority-voting approaches.

Original languageEnglish (US)
Article number6784318
Pages (from-to)667-679
Number of pages13
JournalIEEE Journal on Selected Topics in Signal Processing
Issue number4
StatePublished - Aug 2014


  • Crowdsourcing
  • error-control codes
  • multi-class labeling
  • quality assurance

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'Reliable crowdsourcing for multi-class labeling using coding theory'. Together they form a unique fingerprint.

Cite this