Does confidence reporting from the crowd benefit crowdsourcing performance?

Research output: Chapter in Book/Report/Conference proceedingConference contribution

3 Scopus citations

Abstract

We explore the design of an effective crowdsourcing system for an M-ary classification task. Crowd workers complete simple binary microtasks whose results are aggregated to give the final classification decision. We consider the scenario where the workers have a reject option so that they are allowed to skip microtasks when they are unable to or choose not to respond to binary microtasks. Additionally, the workers report quantized confidence levels when they are able to submit definitive answers. We present an aggregation approach using a weighted majority voting rule, where each worker's response is assigned an optimized weight to maximize crowd's classification performance. We obtain a couterintuitive result that the classification performance does not benefit from workers reporting quantized confidence. Therefore, the crowdsourcing system designer should employ the reject option without requiring confidence reporting.

Original languageEnglish (US)
Title of host publicationProceedings - 2017 2nd International Workshop on Social Sensing, SocialSens 2017 (part of CPS Week)
PublisherAssociation for Computing Machinery, Inc
Pages49-54
Number of pages6
ISBN (Electronic)9781450349772
DOIs
StatePublished - Apr 18 2017
Event2nd International Workshop on Social Sensing, SocialSens 2017 - Pittsburgh, United States
Duration: Apr 21 2017 → …

Other

Other2nd International Workshop on Social Sensing, SocialSens 2017
CountryUnited States
CityPittsburgh
Period4/21/17 → …

Keywords

  • Classification
  • Confidence reporting
  • Crowdsourcing
  • Distributed inference
  • Information fusion
  • Reject option

ASJC Scopus subject areas

  • Signal Processing
  • Computer Networks and Communications

Fingerprint Dive into the research topics of 'Does confidence reporting from the crowd benefit crowdsourcing performance?'. Together they form a unique fingerprint.

  • Cite this

    Li, Q., & Varshney, P. K. (2017). Does confidence reporting from the crowd benefit crowdsourcing performance? In Proceedings - 2017 2nd International Workshop on Social Sensing, SocialSens 2017 (part of CPS Week) (pp. 49-54). Association for Computing Machinery, Inc. https://doi.org/10.1145/3055601.3055607