Multi-Object Classification via Crowdsourcing with a Reject Option

Qunwei Li, Aditya Vempaty, Lav R. Varshney, Pramod K. Varshney

Research output: Contribution to journalArticlepeer-review

18 Scopus citations


Consider designing an effective crowdsourcing system for $M$-ary classification where crowd workers complete simple binary microtasks, which are aggregated to give the final result. We consider the novel scenario where workers have a reject option, so they may skip microtasks they are unable or unwilling to do. For example, in mismatched speech transcription, workers who do not know the language may be unable to respond to microtasks in phonological dimensions outside their categorical perception. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize the crowd's classification performance. We evaluate system performance in both exact and asymptotic forms. Furthermore, we consider the setting where there may be a set of greedy workers that complete microtasks even when they are unable to perform it reliably. We consider an oblivious and an expurgation strategy to deal with greedy workers, developing an algorithm to adaptively switch between the two based on the estimated fraction of greedy workers in the anonymous crowd. Simulation results show improved performance compared with conventional majority voting.

Original languageEnglish (US)
Article number7747496
Pages (from-to)1068-1081
Number of pages14
JournalIEEE Transactions on Signal Processing
Issue number4
StatePublished - Feb 15 2017


  • Classification
  • Reject option
  • crowdsourcing
  • distributed inference
  • information fusion

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'Multi-Object Classification via Crowdsourcing with a Reject Option'. Together they form a unique fingerprint.

Cite this