Commit 544aed6779aa2ce5f340f1e72172c2f3eaad2f38

Authored by waldispuhl
1 parent 8938da05e0
Exists in master

intro

Showing 3 changed files with 73 additions and 12 deletions Side-by-side Diff

CHIpaper/MarketPaper.pdf View file @ 544aed6

No preview for this file type

CHIpaper/MarketPaper.tex View file @ 544aed6
... ... @@ -132,38 +132,35 @@
132 132 challenge system can be used to influence and guide the players towards producing better solutions.
133 133 \end{abstract}
134 134  
135   -\keywords{Human computing; Game; Graph problem; Market; Skills; Challenges}
  135 +\keywords{Human computing; Collaboration; Crowdsourcing; Graph problem; Game; Market; Trading game; Skills; Challenges.}
136 136  
137 137 \section{Introduction}
138 138 Human-computation and crowd-sourcing are now perceived as valuable techniques to help solving difficult computational problems. In order to make the best use of human skills in these systems, it is important to be able to characterize the expertise and performance of humans as individual and even more importantly as groups.
139 139  
140 140 Currently, popular crowd-computing platform such as Amazon Mechanical Turk (AMT) \cite{Buhrmester01012011, Paolacci} or Crowdcrafting \cite{Crowdcrafting} are based on similars divide-and-conquer architectures, where the initial problem is decomposed into smaller sub-tasks that are distributed to individual workers and then aggregated to build a solution. In particular, these systems prevent any interaction between workers in order to prevent groupthink phenomena and bias in the solution \cite{Lorenz:2011aa}.
141 141  
142   -However, such constraints are necessarily limiting the capacity of the system to harness the cognitive power of crowds and make full benefit of collective intelligence. In order to gain expressivity and improve their performance, the next generation of human-computation platforms will undoubtedly need to relax these constraints and build market systems in which workers can collaborate. Nonetheless, before transitioning to this new model, it is important to first estimate the gain of productivity and quantify the usefulness of the mechanisms and incentives to promote collaborative solving and prevent groupthink.
  142 +However, such constraints are necessarily limiting the capacity of the system to harness the cognitive power of crowds and make full benefit of collective intelligence. For instance, iterative combinations of crowdsourced contributions can help enhancing creativity \cite{DBLP:conf/chi/YuN11}. The usefulness of parallelizing workflows has also been suggested for tasks accepting broad varieties of answers \cite{DBLP:conf/chi/Little10}.
143 143  
144   -In \cite{DBLP:conf/chi/Little10}, G. Little \text{et al.} compared the performances of iterative and parallel work processes on AMT.
  144 +The benefits of developing recommendation systems or coordination methods in collaborative environments has been demonstrated \cite{DBLP:conf/cscw/KitturK08,DBLP:conf/cscw/DowKKH12,DBLP:conf/chi/ZhangLMGPH12}. Therefore, in order to gain expressivity and improve their performance, the next generation of human-computation systems will certainly need to implement mechanisms to promote and control the collaboration between workers. Nonetheless, before transitioning to this model, it is important to first estimate the potential gains in productivity, and quantify the usefulness of the mechanisms and incentives to promote collaborative solving and prevent groupthink.
145 145  
146   -%A multiagent perspective is taken to examine the principles of both gameplay and mechanism design for productive games. \cite{DBLP:conf/atal/TsaiLCHH08}
147   -
148   -%Shepherding the crowd yields better work \cite{DBLP:conf/cscw/DowKKH12}
149   -
150   -%Human computation tasks with global constraints \cite{DBLP:conf/chi/ZhangLMGPH12}
151   -
152 146 Historically, computation on graphs has proven to be a good model to study the performance of humans in solving complex combinatorial problems \cite{Kearns:2006aa}. Experiments have been conducted to evaluate the dynamics of crowds collaborating at solving graph problems \cite{DBLP:journals/cacm/Kearns12} but still, little is known about the efficiency of various modes of interaction.
153 147  
154   -In this paper, we propose a formal framework to study human collaborative solving. We design a market system coupled with skills and a challenge system to help the players solve combinatorial graph problems. In order to prevent any bias, we implement this system as a game that makes abstraction of the graphical nature of the underlying problem.
  148 +In this paper, we propose a formal framework to study human collaborative solving. We embed a combinatorial graph problem into a novel multiplayer game-with-a-purpose \cite{DBLP:conf/chi/AhnD04,DBLP:conf/aaai/HoCH07}, which will be used engage participants and analyze collective performances. More precisely, we design a market game in which players can sell and buy their solutions, and coupled this platform with (i) a skills systems to enhance the efficiency of specific gaming strategies and (ii) a challenge systems to guide the work of the crowd. We use this game to investigate the validity of the following hypotheses.
155 149  
156 150 \subsection{Hypotheses}
157 151  
158 152 The development of the game with its three main features, {\em i.e.} the market, the skills and the challenge system, was based on those four hypotheses:
159 153 \begin{enumerate}
  154 +\setlength{\itemsep}{0em}
160 155 \item A market system will help the players build better solutions.
161 156 \item A skill system is useful to orient the players into doing specific actions that are beneficial to the game and other players.
162 157 \item A challenge system is effective in encouraging the players to do a specific action in the game.
163 158 \item The collected solutions are better when all the 3 features are present in a game session, independently of the players' skills.
164 159 \end{enumerate}
165 160  
166   -The goal of the work presented in this paper is to verify if those hypotheses are valid.
  161 +To answer these questions, we conducted a study on 120 participants using different variants of our market game. Our results confirm the benefits of using a trading platform to produce better solutions. Interestingly, we also found that a skills system helps to promote actions favorable to the collective solving process (e.g. increasing the diversity of intermediate solutions), but that the efficiency of a skill is reduced if it is designed to solve one of the primary objectives of the game. Finally, we observed that a precise parametrization of challenges (i.e. finding an appropriate difficulty, nor too easy, nor too difficult) is required to result in an improvement of the quality of the collective work.
  162 +
  163 +Our game is freely available at \texttt{URL:TBA}, and can be used as a platform for further independent studies.
167 164  
168 165 \section{Problem}
169 166  
CHIpaper/references.bib View file @ 544aed6
... ... @@ -2,12 +2,76 @@
2 2 %% http://bibdesk.sourceforge.net/
3 3  
4 4  
5   -%% Created for Jerome Waldispuhl at 2015-09-24 18:15:20 -0400
  5 +%% Created for Jerome Waldispuhl at 2015-09-25 01:30:22 -0400
6 6  
7 7  
8 8 %% Saved with string encoding Unicode (UTF-8)
9 9  
10 10  
  11 +
  12 +@inproceedings{DBLP:conf/aaai/HoCH07,
  13 + Author = {Chien{-}Ju Ho and Tsung{-}Hsiang Chang and Jane Yung{-}jen Hsu},
  14 + Bibsource = {dblp computer science bibliography, http://dblp.org},
  15 + Biburl = {http://dblp.uni-trier.de/rec/bib/conf/aaai/HoCH07},
  16 + Booktitle = {Proceedings of the Twenty-Second {AAAI} Conference on Artificial Intelligence, July 22-26, 2007, Vancouver, British Columbia, Canada},
  17 + Date-Added = {2015-09-25 05:30:10 +0000},
  18 + Date-Modified = {2015-09-25 05:30:21 +0000},
  19 + Pages = {1359--1364},
  20 + Timestamp = {Mon, 10 Dec 2012 15:34:43 +0100},
  21 + Title = {PhotoSlap: {A} Multi-player Online Game for Semantic Annotation},
  22 + Url = {http://www.aaai.org/Library/AAAI/2007/aaai07-215.php},
  23 + Year = {2007},
  24 + Bdsk-Url-1 = {http://www.aaai.org/Library/AAAI/2007/aaai07-215.php}}
  25 +
  26 +@inproceedings{DBLP:conf/chi/AhnD04,
  27 + Abstract = {We introduce a new interactive system: a game that is fun and can be used to create valuable output. When people play the game they help determine the contents of images by providing meaningful labels for them. If the game is played as much as popular online games, we estimate that most images on the Web can be labeled in a few months. Having proper labels associated with each image on the Web would allow for more accurate image search, improve the accessibility of sites (by providing descriptions of images to visually impaired individuals), and help users block inappropriate images. Our system makes a significant contribution because of its valuable output and because of the way it addresses the image-labeling problem. Rather than using computer vision techniques, which don't work well enough, we encourage people to do the work by taking advantage of their desire to be entertained.},
  28 + Author = {Luis von Ahn and Laura Dabbish},
  29 + Bibsource = {dblp computer science bibliography, http://dblp.org},
  30 + Biburl = {http://dblp.uni-trier.de/rec/bib/conf/chi/AhnD04},
  31 + Booktitle = {Proceedings of the 2004 Conference on Human Factors in Computing Systems, {CHI} 2004, Vienna, Austria, April 24 - 29, 2004},
  32 + Date-Added = {2015-09-25 05:13:24 +0000},
  33 + Date-Modified = {2015-09-25 05:13:38 +0000},
  34 + Doi = {10.1145/985692.985733},
  35 + Pages = {319--326},
  36 + Timestamp = {Fri, 10 Feb 2006 16:00:37 +0100},
  37 + Title = {Labeling images with a computer game},
  38 + Url = {http://doi.acm.org/10.1145/985692.985733},
  39 + Year = {2004},
  40 + Bdsk-Url-1 = {http://doi.acm.org/10.1145/985692.985733},
  41 + Bdsk-Url-2 = {http://dx.doi.org/10.1145/985692.985733}}
  42 +
  43 +@inproceedings{DBLP:conf/chi/YuN11,
  44 + Author = {Lixiu Yu and Jeffrey V. Nickerson},
  45 + Bibsource = {dblp computer science bibliography, http://dblp.org},
  46 + Biburl = {http://dblp.uni-trier.de/rec/bib/conf/chi/YuN11},
  47 + Booktitle = {Proceedings of the International Conference on Human Factors in Computing Systems, {CHI} 2011, Vancouver, BC, Canada, May 7-12, 2011},
  48 + Date-Added = {2015-09-25 04:13:04 +0000},
  49 + Date-Modified = {2015-09-25 04:13:48 +0000},
  50 + Doi = {10.1145/1978942.1979147},
  51 + Pages = {1393--1402},
  52 + Timestamp = {Wed, 11 May 2011 10:43:31 +0200},
  53 + Title = {Cooks or cobblers?: crowd creativity through combination},
  54 + Url = {http://doi.acm.org/10.1145/1978942.1979147},
  55 + Year = {2011},
  56 + Bdsk-Url-1 = {http://doi.acm.org/10.1145/1978942.1979147},
  57 + Bdsk-Url-2 = {http://dx.doi.org/10.1145/1978942.1979147}}
  58 +
  59 +@inproceedings{DBLP:conf/cscw/KitturK08,
  60 + Abstract = {Wikipedia's success is often attributed to the large numbers of contributors who improve the accuracy, completeness and clarity of articles while reducing bias. However, because of the coordination needed to write an article collaboratively, adding contributors is costly. We examined how the number of editors in Wikipedia and the coordination methods they use affect article quality. We distinguish between explicit coordination, in which editors plan the article through communication, and implicit coordination, in which a subset of editors structure the work by doing the majority of it. Adding more editors to an article improved article quality only when they used appropriate coordination techniques and was harmful when they did not. Implicit coordination through concentrating the work was more helpful when many editors contributed, but explicit coordination through communication was not. Both types of coordination improved quality more when an article was in a formative stage. These results demonstrate the critical importance of coordination in effectively harnessing the wisdom of the crowd in online production environments.},
  61 + Author = {Aniket Kittur and Robert E. Kraut},
  62 + Bibsource = {dblp computer science bibliography, http://dblp.org},
  63 + Biburl = {http://dblp.uni-trier.de/rec/bib/conf/cscw/KitturK08},
  64 + Booktitle = {Proceedings of the 2008 {ACM} Conference on Computer Supported Cooperative Work, {CSCW} 2008, San Diego, CA, USA, November 8-12, 2008},
  65 + Date-Added = {2015-09-25 03:07:08 +0000},
  66 + Date-Modified = {2015-09-25 03:07:27 +0000},
  67 + Doi = {10.1145/1460563.1460572},
  68 + Pages = {37--46},
  69 + Timestamp = {Mon, 24 Nov 2008 10:58:36 +0100},
  70 + Title = {Harnessing the wisdom of crowds in wikipedia: quality through coordination},
  71 + Url = {http://doi.acm.org/10.1145/1460563.1460572},
  72 + Year = {2008},
  73 + Bdsk-Url-1 = {http://doi.acm.org/10.1145/1460563.1460572},
  74 + Bdsk-Url-2 = {http://dx.doi.org/10.1145/1460563.1460572}}
11 75  
12 76 @inproceedings{DBLP:conf/chi/ZhangLMGPH12,
13 77 Author = {Haoqi Zhang and Edith Law and Rob Miller and Krzysztof Gajos and David C. Parkes and Eric Horvitz},