Scheduling human intelligence tasks in multi-tenant crowd-powered systems

Difallah, Djellel Eddine, Demartini, Gianluca and Cudré-Mauroux, Philippe (2016). Scheduling human intelligence tasks in multi-tenant crowd-powered systems. In: 25th International World Wide Web Conference, WWW 2016. 25th International World Wide Web Conference, WWW 2016, Montreal, QC, Canada, (855-865). 11-15 April 2016. doi:10.1145/2872427.2883030


Author Difallah, Djellel Eddine
Demartini, Gianluca
Cudré-Mauroux, Philippe
Title of paper Scheduling human intelligence tasks in multi-tenant crowd-powered systems
Conference name 25th International World Wide Web Conference, WWW 2016
Conference location Montreal, QC, Canada
Conference dates 11-15 April 2016
Proceedings title 25th International World Wide Web Conference, WWW 2016
Place of Publication Geneva, Switzerland
Publisher International World Wide Web Conferences Steering Committee
Publication Year 2016
Sub-type Fully published paper
DOI 10.1145/2872427.2883030
Open Access Status Not yet assessed
ISBN 9781450341431
Start page 855
End page 865
Total pages 11
Language eng
Abstract/Summary Micro-Task crowdsourcing has become a popular approach to efiectively tackle complex data management problems such as data linkage, missing values, or schema matching. However, the backend crowdsourced operators of crowd-powered systems typically yield higher latencies than the machineprocessable operators, this is mainly due to inherent efficiency difierences between humans and machines. This problem can be further exacerbated by the lack of workers on the target crowdsourcing platform, or when the workers are shared unequally among a number of competing requesters; including the concurrent users from the same organization who execute crowdsourced queries with difierent types, priorities and prices. Under such conditions, a crowd-powered system acts mostly as a proxy to the crowdsourcing platform, and hence it is very difficult to provide effiency guarantees to its end-users. Scheduling is the traditional way of tackling such problems in computer science, by prioritizing access to shared resources. In this paper, we propose a new crowdsourcing system architecture that leverages scheduling algorithms to optimize task execution in a shared resources environment, in this case a crowdsourcing platform. Our study aims at assessing the efficiency of the crowd in settings where multiple types of tasks are run concurrently. We present extensive experimental results comparing i) difierent multi-Tenant crowdsourcing jobs, including a workload derived from real traces, and ii) difierent scheduling techniques tested with real crowd workers. Our experimental results show that task scheduling can be leveraged to achieve fairness and reduce query latency in multi-Tenant crowd-powered systems, although with very different tradeoffs compared to traditional settings not including human factors.
Subjects 1705 Computer Networks and Communications
1712 Software
Keyword Crowd-Powered System
Crowdsourcing
Scheduling
Q-Index Code E1
Q-Index Status Provisional Code
Grant ID PP00P2 153023
Institutional Status Non-UQ

Document type: Conference Paper
Sub-type: Fully published paper
Collection: School of Information Technology and Electrical Engineering Publications
 
Versions
Version Filter Type
Citation counts: Scopus Citation Count Cited 4 times in Scopus Article | Citations
Google Scholar Search Google Scholar
Created: Thu, 26 Apr 2018, 15:55:29 EST by Gianluca Demartini on behalf of Learning and Research Services (UQ Library)