Commit 4adfe24734b2531b2e35e19fb83d54716d56a135

Authored by Olivier
1 parent 544aed6779
Exists in master

Final touches. Small tweaks all over the paper. This is the version that was submitted to CHI2016.

Showing 2 changed files with 75 additions and 61 deletions Side-by-side Diff

CHIpaper/MarketPaper.pdf View file @ 4adfe24

No preview for this file type

CHIpaper/MarketPaper.tex View file @ 4adfe24
... ... @@ -103,7 +103,7 @@
103 103 % End of preamble. Here it comes the document.
104 104 \begin{document}
105 105  
106   -\title{Collaborative solving in a human computing game using a market, skills and challenges}
  106 +\title{Collaborative Solving in a Human Computing Game Using a Market, Skills and Challenges}
107 107  
108 108 \numberofauthors{3}
109 109 \author{%
... ... @@ -125,8 +125,8 @@
125 125  
126 126 \begin{abstract}
127 127 Using a human computing game to solve a problem that has a large search space is not straightforward. The difficulty of using such an approach
128   -comes from the following facts: (1) it would be overwhelming for a single player to show him or her the complete search space and at the same time,
129   -(2) it is impossible to find an optimal solution without considering all the available data. In this paper, we present a human computing
  128 +comes from the following facts: (i) it would be overwhelming for a single player to show him or her the complete search space and at the same time,
  129 +(ii) it is impossible to find an optimal solution without considering all the available data. In this paper, we present a human computing
130 130 game that uses a market, skills and a challenge system to help the players solve a graph problem in a collaborative manner. The results obtained
131 131 during 12 game sessions of 10 players show that the market helps players to build larger solutions. We also show that a skill system and, to a lesser extent, a
132 132 challenge system can be used to influence and guide the players towards producing better solutions.
133 133  
134 134  
135 135  
... ... @@ -135,17 +135,17 @@
135 135 \keywords{Human computing; Collaboration; Crowdsourcing; Graph problem; Game; Market; Trading game; Skills; Challenges.}
136 136  
137 137 \section{Introduction}
138   -Human-computation and crowd-sourcing are now perceived as valuable techniques to help solving difficult computational problems. In order to make the best use of human skills in these systems, it is important to be able to characterize the expertise and performance of humans as individual and even more importantly as groups.
  138 +Human-computation and crowd-sourcing are now perceived as valuable techniques to help solving difficult computational problems. In order to make the best use of human skills in these systems, it is important to be able to characterize the expertise and performance of humans as individuals and even more importantly as groups.
139 139  
140   -Currently, popular crowd-computing platform such as Amazon Mechanical Turk (AMT) \cite{Buhrmester01012011, Paolacci} or Crowdcrafting \cite{Crowdcrafting} are based on similars divide-and-conquer architectures, where the initial problem is decomposed into smaller sub-tasks that are distributed to individual workers and then aggregated to build a solution. In particular, these systems prevent any interaction between workers in order to prevent groupthink phenomena and bias in the solution \cite{Lorenz:2011aa}.
  140 +Currently, popular crowd-computing platforms such as Amazon Mechanical Turk (AMT) \cite{Buhrmester01012011, Paolacci} or Crowdcrafting \cite{Crowdcrafting} are based on similar divide-and-conquer architectures, where the initial problem is decomposed into smaller sub-tasks that are distributed to individual workers and then aggregated to build a solution. In particular, these systems prevent any interaction between workers in order to prevent groupthink phenomena and bias in the solution \cite{Lorenz:2011aa}.
141 141  
142 142 However, such constraints are necessarily limiting the capacity of the system to harness the cognitive power of crowds and make full benefit of collective intelligence. For instance, iterative combinations of crowdsourced contributions can help enhancing creativity \cite{DBLP:conf/chi/YuN11}. The usefulness of parallelizing workflows has also been suggested for tasks accepting broad varieties of answers \cite{DBLP:conf/chi/Little10}.
143 143  
144 144 The benefits of developing recommendation systems or coordination methods in collaborative environments has been demonstrated \cite{DBLP:conf/cscw/KitturK08,DBLP:conf/cscw/DowKKH12,DBLP:conf/chi/ZhangLMGPH12}. Therefore, in order to gain expressivity and improve their performance, the next generation of human-computation systems will certainly need to implement mechanisms to promote and control the collaboration between workers. Nonetheless, before transitioning to this model, it is important to first estimate the potential gains in productivity, and quantify the usefulness of the mechanisms and incentives to promote collaborative solving and prevent groupthink.
145 145  
146   -Historically, computation on graphs has proven to be a good model to study the performance of humans in solving complex combinatorial problems \cite{Kearns:2006aa}. Experiments have been conducted to evaluate the dynamics of crowds collaborating at solving graph problems \cite{DBLP:journals/cacm/Kearns12} but still, little is known about the efficiency of various modes of interaction.
  146 +Historically, computation on graphs has proven to be a good model to study the performance of humans in solving complex combinatorial problems \cite{Kearns:2006aa}. Experiments have been conducted to evaluate the dynamics of crowds collaborating at solving graph problems \cite{DBLP:journals/cacm/Kearns12} but still, little is known about the efficiency of the various modes of interaction.
147 147  
148   -In this paper, we propose a formal framework to study human collaborative solving. We embed a combinatorial graph problem into a novel multiplayer game-with-a-purpose \cite{DBLP:conf/chi/AhnD04,DBLP:conf/aaai/HoCH07}, which will be used engage participants and analyze collective performances. More precisely, we design a market game in which players can sell and buy their solutions, and coupled this platform with (i) a skills systems to enhance the efficiency of specific gaming strategies and (ii) a challenge systems to guide the work of the crowd. We use this game to investigate the validity of the following hypotheses.
  148 +In this paper, we propose a formal framework to study human collaborative solving. We embed a combinatorial graph problem into a novel multiplayer game-with-a-purpose \cite{DBLP:conf/chi/AhnD04,DBLP:conf/aaai/HoCH07}, which will be used to engage participants and analyze collective performances. More precisely, we design a market game in which players can sell and buy solutions or bits of information, and couple this platform with (i) a skills system to enhance the efficiency of specific gaming strategies and (ii) a challenge systems to guide the work of the crowd. We use this game to investigate the validity of the following hypotheses.
149 149  
150 150 \subsection{Hypotheses}
151 151  
152 152  
... ... @@ -155,10 +155,12 @@
155 155 \item A market system will help the players build better solutions.
156 156 \item A skill system is useful to orient the players into doing specific actions that are beneficial to the game and other players.
157 157 \item A challenge system is effective in encouraging the players to do a specific action in the game.
158   - \item The collected solutions are better when all the 3 features are present in a game session, independently of the players' skills.
  158 + \item The collected solutions are better when all the three features are present in a game session, independently of the players' personal skills.
159 159 \end{enumerate}
160 160  
161   -To answer these questions, we conducted a study on 120 participants using different variants of our market game. Our results confirm the benefits of using a trading platform to produce better solutions. Interestingly, we also found that a skills system helps to promote actions favorable to the collective solving process (e.g. increasing the diversity of intermediate solutions), but that the efficiency of a skill is reduced if it is designed to solve one of the primary objectives of the game. Finally, we observed that a precise parametrization of challenges (i.e. finding an appropriate difficulty, nor too easy, nor too difficult) is required to result in an improvement of the quality of the collective work.
  161 +To answer these questions, we conducted a study on 120 participants using different variants of our market game. Our results confirm the benefits of using a trading platform to produce better solutions. Interestingly, we also found that a skills system helps to promote actions that are favorable to the collective solving process,
  162 +%(e.g. increasing the diversity of intermediate solutions)
  163 +but that the efficiency of a skill is reduced if it is designed to help solve one of the primary objectives of the game. Finally, we observed that a precise parametrization of challenges (i.e. finding an appropriate difficulty, nor too easy, nor too difficult) is required to result in an improvement of the quality of the collective work.
162 164  
163 165 Our game is freely available at \texttt{URL:TBA}, and can be used as a platform for further independent studies.
164 166  
165 167  
... ... @@ -179,13 +181,19 @@
179 181 colors of the vertices, it is possible to show the players only the colored vertices. To solve the problem, the players have to find the largests sets
180 182 of circles with colors in common, for all possible subsets of colors.
181 183  
  184 +Note that it is not our goal to compare the performance of players with the performance of computers in solving this problem. With a limited number of colors
  185 +(like six in our tests), the exact algorithm can solve the problem in seconds. For this study, we required a
  186 +problem that was structured enough so that we could
  187 +easily calculate the optimal solution and evaluate the performance of the players depending on what features were on or off and also the effect of the different
  188 +features on the quality of solutions.
182 189  
183 190  
184 191 \section{Presentation of the game}
185 192  
186 193 \subsection{Goal of the game}
187 194  
188   -The main objective of the game is to build {\em sequences} ({\em i.e.} sets) of circles that are as long as possible and contain as many colors in common
  195 +The main objective of the game is to build {\em sequences} ({\em i.e.} sets) of circles (circles represent vertices of the graph) that (i) are as long as possible
  196 +and (ii) contain as many colors in common
189 197 as possible. Circles used by the players to build the sequences either come random packages bought from the system or they come from another player
190 198 through the market. The sequences can then be sold to the system for a certain amount of game money, which is determined by a scoring function that takes into
191 199 account the length and the number of colors in common of the sequence.
192 200  
... ... @@ -198,10 +206,10 @@
198 206 that is proportional to the difficulty of building the sequence. More precisely,
199 207 we calculated the average length $L_n$ of all solutions for each $n$ number of colors. The base score is simply the reciprocal of this average ($1/L_n$) multiplied
200 208 by a balancing factor (505 in our case). The balancing factor was chosen in order to get a score of 500 for a sequence of length 10 with only one color in common,
201   -which is exactly the price of two random packages of circles.
  209 +which is exactly the price of two random packages of circles. Also notice that the value of a sequence is exponential in relation to its length, which is
  210 +to encourage players to build the longest possible sequences.
202 211  
203 212 \begin{table}[h]
204   -\caption{Value of the base score depending on the number of colors in common}\label{tab_baseScore}
205 213 \begin{center}
206 214 \begin{tabular}{cc}\hline
207 215 Number of colors & Base score\\
... ... @@ -214,6 +222,7 @@
214 222 6 & 72\\\hline
215 223 \end{tabular}
216 224 \end{center}
  225 +\caption{Value of the base score depending on the number of colors in common}\label{tab_baseScore}
217 226 \end{table}
218 227  
219 228 \subsection{Game interface}
220 229  
221 230  
222 231  
... ... @@ -221,21 +230,21 @@
221 230 The game client and the server were built in Java 1.7.
222 231 As shown in Figure~\ref{fig_interface}, the game interface can be divided into 3 parts: the player information panel, the game panel and the market panel.
223 232  
224   -\subsubsection{Player information panel}
  233 +\subsubsection{A: Player information panel}
225 234  
226 235 This panel simply contains information on the player's wallet, the current level of the player and has three buttons, allowing the player to open
227 236 dialogs showing information on the current challenge, the skills (see section Skills for a description of the available skills) and the leaderboard.
228 237 One experience point is given to the player for each game dollar that he/she wins. The player can lose game money, but cannot lose experience points
229   -(experience points can only go up).
  238 +(experience points can only go up). Every time a player levels-up, he/she gets a skill point that can be used to improve any of the skills.
230 239  
231   -\subsubsection{Game panel}
  240 +\subsubsection{B: Game panel}
232 241  
233 242 The first component of the game panel is the 'My sequence' panel, which shows the current sequence that is being built by the player. The maximum size
234 243 of a sequence is 10. Colors in common in the
235 244 sequence are indicated by a thick black border surrounding the colors in the circles. Players can use the arrows to switch between the different sequence slots
236 245 (2 sequence slots are available at the start of the game). The current value of the sequence is shown at the right, and the price for adding one more circle
237 246 with the same colors in common is shown right below in gray. Finally, the sell button allows the player to sell the current sequence to the system: the sequence then
238   -disappears and the money is given to the player.
  247 +disappears and the money is given to the player. Selling a sequence is equivalent to submitting a solution to the system.
239 248  
240 249 The second component is the 'My hand' panel, which can contain up to 20 circles. Players can add a circle to the sequence by clicking on it. Circles are
241 250 represented by their colors and by a price label (in a black box). The price corresponds to the current value of the circle on the market. Clicking on
... ... @@ -248,7 +257,7 @@
248 257 Finally, the bottom panel is a news feed, showing information on the game state, like the remaining time to complete the challenge and the last transactions
249 258 completed by the player for example.
250 259  
251   -\subsubsection{Market panel}
  260 +\subsubsection{C: Market panel}
252 261  
253 262 At the top of the market panel, buttons allow the player to create bids for circles or to buy random packages (or bags) of circles.
254 263 The 'Random bag' costs \$250 and contains 5 circles with fewer colors. The 'Premium bag' costs \$500 and contains 5 circles
255 264  
... ... @@ -256,11 +265,13 @@
256 265  
257 266 Right below the buttons is the 'Automatic bids' panel, which allows the player to get automatic bids for circles corresponding to the sequences that
258 267 he or she is building. A percentage of profit for the price of the automatic bids can be set with the slider.
259   -The profit is defined as the money the player would make by adding one more SNP with the same colors in the current sequence (difference between the gray and black prices).
  268 +The profit is defined as the money the player would make by adding one more circle with the same colors in the current sequence (difference between the gray and black prices
  269 +above the Sell button).
260 270  
261 271 The 'My bids' panel shows all the bids that the player currently has on the market. The bid price is shown below the circle (in the black box). On the right side
262 272 of the circle is the number of sequences with the same colors that the player can buy from other players (in the blue box).
263   -Clicking on the blue box opens a window showing the list of sequences that can be bought. Buying a sequence from another player is called a 'buyout'.
  273 +Clicking on the blue box opens a window showing the list of sequences that can be bought. Buying a sequence from another player is called a 'buyout' (see
  274 +the following subsection for a more detailed description of buyouts).
264 275  
265 276 Finally, the last panel at the bottom shows the last circle or sequence that was bought by the player.
266 277  
... ... @@ -276,8 +287,8 @@
276 287  
277 288 \subsection{Market}
278 289  
279   -The market has three functions: allow the players to exchange circles through a bidding system, allow players to buy sequences built and sold by other
280   -players so that they can be improved, and merge together sequences of length 10 to create super circles that are then put back in the game.
  290 +The market has three functions: (i) allow the players to exchange circles through a bidding system, (ii) allow players to buy sequences built and sold by other
  291 +players so that they can be improved, and (iii) merge together sequences of length 10 to create super circles that are then put back in the game.
281 292  
282 293 For every subset of colors, the server has a list of all the bids that are currently on the market. The value of
283 294 the highest bid on the market is shown below every circle under the possession of the players. When a circle is sold by a player, it is sent through
... ... @@ -289,7 +300,7 @@
289 300  
290 301 Finally, the game system creates a super circle every time a sequence of length 10 is sold by a player. A super circle of level 2 (representing 10 circles)
291 302 counts as two circles when put in a sequence. Super circles can be of any level (a sequence of 10 super circles of level 2 form a super circle of level 3, and so on).
292   -The objective is to remove the limitation of the maximum sequence size imposed by the game interface.
  303 +The idea behind the creation of the super circles was to remove the limitation of the maximum sequence size imposed by the game interface.
293 304  
294 305 \subsection{Skills}
295 306  
... ... @@ -298,10 +309,10 @@
298 309 guide the player in doing actions that are beneficial to the system or to the other players:
299 310  
300 311 \begin{itemize}
301   -\item {\em Buyout King}: lowers the price of buying a sequence from another player;
302   -\item {\em Color Expert}: gives a bonus to selling sequences that have more than one color in common;
303   -\item {\em Sequence Collector}: gives an additional sequence slot;
304   -\item {\em Master Trader}: gives a bonus to selling SNPs to other players.
  312 +\item {\em Buyout King}: lowers the price of buying a sequence from another player (goal: encourage buyouts);
  313 +\item {\em Color Expert}: gives a bonus to selling sequences that have more than one color in common (goal: push players to build more multicolored sequences);
  314 +\item {\em Sequence Collector}: gives an additional sequence slot (goal: give more space to encourage the creation of longer sequences with more colors in common);
  315 +\item {\em Master Trader}: gives a bonus to selling circles to other players (goal: promote the selling of individual circles).
305 316 \end{itemize}
306 317  
307 318 \subsection{Challenge system}
... ... @@ -319,7 +330,7 @@
319 330  
320 331 Basically, the system continuously monitors the activities of the players and decreases or increases the probabilities of each challenge type.
321 332 The next challenge is then selected using a multinomial sampling on these probabilities. The number of times $T$ that the challenge-related action must be
322   -completed is selected randomly between 2 and 4. The prize that is awarded for completing the challenge is equal to $1500 * T$.
  333 +completed is selected randomly between 2 and 4. The prize that is awarded for completing the challenge is equal to \$$1500 * T$.
323 334  
324 335 \section{Experiments}
325 336  
... ... @@ -391,7 +402,7 @@
391 402  
392 403 \subsection{Testing hypothesis 1: the efficiency of the market}
393 404  
394   -The market system we implemented in the game allows the players to exchange circles and partial solutions (sequences). The main goal of the market
  405 +The market system we implemented in the game allows the players to exchange circles and partial solutions (in the form of buyouts). The main goal of the market
395 406 is to help the players in building longer sequences.
396 407  
397 408 %\begin{tabular}{ccc}\hline
398 409  
399 410  
... ... @@ -430,11 +441,11 @@
430 441 were shown to be significantly different ($p < 0.01$), except a few shown in table~\ref{tab_Dunn}. Note that the strongest similarities are found between
431 442 the three 'All' groups and the three 'No market' groups. Some of the 'No skills' experiments are found to be similar to the 'All' groups, which could indicate
432 443 that the presence of the skills have a very limited effect on the sequence length. The NC experiment is found to be similar to two 'No market' groups, but that
433   -can be explained by the fact the players for the NC experiment were very weak (see Section Testing hypothesis 4).
  444 +can be explained by the fact the players for the NC experiment were very weak (as can be seen by the total experience gained during that session in
  445 +Figure~\ref{fig_totalXP}).
434 446  
  447 +\setlength{\tabcolsep}{4pt}
435 448 \begin{table}[h]
436   - \caption{Similar groups of sequence length distributions, as reported by Dunn's test. An 'n' in the table represent a similar pair when not considering
437   - super circles, and an 's' in the table represents a similar pair when considering super circles.}\label{tab_Dunn}
438 449 \begin{center}
439 450 \begin{tabular}{c|cccccccc}\hline
440 451 & A-2 & A-3 & NS & NS-2 & NS-3 & NM & NM-2 & NM-3\\\hline
... ... @@ -446,6 +457,8 @@
446 457 NM & & & & & & & n/s & n \\\hline
447 458 \end{tabular}
448 459 \end{center}
  460 +\caption{Similar groups of sequence length distributions, as reported by Dunn's test. An 'n' in the table represents a similar pair when not considering
  461 + super circles, and an 's' in the table represents a similar pair when considering super circles.}\label{tab_Dunn}
449 462 \end{table}
450 463  
451 464 %WILL HAVE TO MOVE THE FOLLOWING SENTENCES TO HYPOTHESIS 4 SECTION
... ... @@ -455,8 +468,8 @@
455 468  
456 469 \subsection{Testing hypothesis 2: the benefits of a skill system}
457 470  
458   -We implemented the skill system for two reasons: (1) to give the players more incentive to accumulate experience points as fast as possible, because
459   -the reward for leveling-up is an additional skill point, and (2) to influence indirectly
  471 +We implemented the skill system for two reasons: (i) to give the players more incentive to accumulate experience points as fast as possible, because
  472 +the reward for leveling-up is an additional skill point, and (ii) to influence indirectly
460 473 the players into doing actions that are either improving the solutions collected by the system or helpful to the other players (which in the end will also
461 474 improve the solutions). In our game, two skills were related to the market ({\em Buyout King} and {\em Master Trader}) and two skills were related to building
462 475 sequences ({\em Color Expert} and {\em Sequence Collector}). In the following paragraphs, we will analyze how those four skills affected the strategies and actions
... ... @@ -473,7 +486,7 @@
473 486  
474 487 \begin{figure}[htbp]
475 488 \begin{center}
476   - \includegraphics[width=\halfWidth]{Figs/boxplot_BK.pdf}
  489 + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_BK.pdf}
477 490 \vspace{0cm}
478 491 \caption{Boxplot of the number of buyouts made by players with (37 players) and without (66 players) the {\em Buyout King} skill.
479 492 }\label{fig_boxplotBK}
... ... @@ -497,7 +510,7 @@
497 510  
498 511 \begin{figure}[htbp]
499 512 \begin{center}
500   - \includegraphics[width=\halfWidth]{Figs/boxplot_MT.pdf}
  513 + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_MT.pdf}
501 514 \vspace{0cm}
502 515 \caption{Boxplot of the number of circles sold individually by players with (33 players) and without (70 players) the {\em Master Trader} skill.
503 516 }\label{fig_boxplotMT}
504 517  
... ... @@ -517,11 +530,11 @@
517 530  
518 531 The {\em Color Expert} skill gives a bonus multiplier to the scoring function for sequences with more than one color in common. This skill was implemented in
519 532 order to give extra motivation to build sequences with many colors in common, since they are harder to build. Indeed, more focus is needed from the player to match
520   -many cicles with more than one color in common.
  533 +many circles with more than one color in common.
521 534  
522 535 \begin{figure}[htbp]
523 536 \begin{center}
524   - \includegraphics[width=\halfWidth]{Figs/boxplot_CE.pdf}
  537 + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_CE.pdf}
525 538 \vspace{0cm}
526 539 \caption{Boxplot of the proportion of sequences with more than one color in common sold by players with (94 players) and without (49 players) the {\em Color Expert} skill.
527 540 }\label{fig_boxplotCE}
... ... @@ -545,7 +558,7 @@
545 558  
546 559 \begin{figure}[htbp]
547 560 \begin{center}
548   - \includegraphics[width=\halfWidth]{Figs/boxplot_SC_seqLength.pdf}
  561 + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_SC_seqLength.pdf}
549 562 \vspace{0cm}
550 563 \caption{Boxplot of the average sequence length of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill.
551 564 }\label{fig_boxplotSC_seqLength}
... ... @@ -553,7 +566,7 @@
553 566 \end{figure}
554 567  
555 568 We first compared the average sequence length of sequences built by players with the {\em Sequence Collector} skill and the ones built by the rest of
556   -players~\ref{fig_boxplotSC_seqLength}. While the median value for the players without the skill (5.63) is a little bit larger than the one for the players
  569 +players (see Figure~\ref{fig_boxplotSC_seqLength}). While the median value for the players without the skill (5.63) is a little bit larger than the one for the players
557 570 with the skill (5.12), the averages of both groups are actually similar (5.61 and 5.56 in the same order). Since the distribution of the average
558 571 sequence lengths were not normal (the Shapiro-Wilk test rejected the null hypothesis with $p = 0.0057$), we did a Mann Whitney's U test to compare the
559 572 medians of both groups. The test failed to reject the null hypothesis that the values were sampled from the same distribution ($p = 0.69$). Thus,
560 573  
... ... @@ -564,15 +577,15 @@
564 577  
565 578 \begin{figure}[htbp]
566 579 \begin{center}
567   - \includegraphics[width=\halfWidth]{Figs/boxplot_SC_nbCols.pdf}
  580 + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_SC_nbCols.pdf}
568 581 \vspace{0cm}
569 582 \caption{Boxplot of the average number of colors in common of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill.
570 583 }\label{fig_boxplotSC_nbCols}
571 584 \end{center}
572 585 \end{figure}
573 586  
574   -We then compared the average number of colors in common of the sequences built by players with and without the {\em Sequence collector} skill~\ref{fig_boxplotSC_nbCols}.
575   -The median value for the players without the skill (1.58) is 14\% lower than the one for the players with the skill (1.80). Since the distribution of the average
  587 +We then compared the average number of colors in common of the sequences built by players with and without the {\em Sequence collector} skill (see Figure~\ref{fig_boxplotSC_nbCols}).
  588 +The median value for the players without the skill (1.58) is 12\% lower than the one for the players with the skill (1.80). Since the distribution of the average
576 589 number of colors in common is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis
577 590 with $p = 1.2E-7$), we used a Mann-Whitney's U test to compare the medians of the two groups and we found a significant effect of the presence
578 591 of this skill on the medians ($U = 3113$, $p = 0.01$, effect size $r = 0.21$). The {\em Sequence collector} skill is thus helping players
... ... @@ -600,7 +613,7 @@
600 613 \vspace{0cm}
601 614 \caption{Average number of colors in the sequences with and without the {\em Minimum number of colors challenge} active. 'A', 'A-2' and 'A-3'
602 615 represent the tests with all the features present, 'NS', 'NS-2' and 'NS-3' represent the tests without the skills, and 'NM', 'NM-2' and 'NM-3' represents
603   - the test without the market.
  616 + the tests without the market.
604 617 }\label{fig_minNbCols}
605 618 \end{center}
606 619 \end{figure}
607 620  
608 621  
... ... @@ -620,14 +633,14 @@
620 633 \begin{center}
621 634 \includegraphics[width=\halfWidth]{Figs/minSeqLength.pdf}
622 635 \vspace{0cm}
623   - \caption{Average sequence length with and without the {\em Minimum sequence length challenge active}. 'A', 'A-2' and 'A-3'
  636 + \caption{Average sequence length with and without the {\em Minimum sequence length challenge} active. 'A', 'A-2' and 'A-3'
624 637 represent the tests with all the features present, 'NS', 'NS-2' and 'NS-3' represent the tests without the skills, and 'NM', 'NM-2' and 'NM-3' represents
625   - the test without the market.
  638 + the tests without the market.
626 639 }\label{fig_minSeqLength}
627 640 \end{center}
628 641 \end{figure}
629 642  
630   -The means of the average sequence lengths during the challenge and for the rest of the time are 5.38 and 5.08 respectively. Since the distribution
  643 +The means of all the average sequence lengths during the challenge and for the rest of the time are 5.38 and 5.08 respectively. Since the distribution
631 644 of the averages of sequence lengths is normal (Shapiro-Wilk $p = 0.27$), we used a Welch's t-test to compare those means, but the test wasn't able
632 645 to prove that those means are significantly different ($t(16)=0.79$, $p = 0.44$).
633 646  
... ... @@ -645,9 +658,8 @@
645 658 \begin{center}
646 659 \includegraphics[width=\halfWidth]{Figs/sellBuySNP.pdf}
647 660 \vspace{0cm}
648   - \caption{Number of individual circles sold on the market per minute with and without the Sell/buy challenge active. 'A', 'A-2' and 'A-3'
649   - represent the tests with all the features present, 'NS', 'NS-2' and 'NS-3' represent the tests without the skills, and 'NM', 'NM-2' and 'NM-3' represents
650   - the test without the market.
  661 + \caption{Number of individual circles sold on the market per minute with and without the {\em Sell/buy challenge} active. 'A', 'A-2' and 'A-3'
  662 + represent the tests with all the features present, and 'NS', 'NS-2' and 'NS-3' represent the tests without the skills.
651 663 }\label{fig_sellBuySNP}
652 664 \end{center}
653 665 \end{figure}
... ... @@ -679,7 +691,7 @@
679 691 by the players in each of the game sessions. Our initial hypothesis was that players who have access to all the game features should
680 692 be able to solve more of the problem.
681 693  
682   -Interestingly, we observed a larger than expected variance in the participants' skills which made it sometimes
  694 +Interestingly, we observed a larger than expected variance in the participants' personal skills which made it sometimes
683 695 difficult to compare one game session with another in terms of the percentage of the problem that was solved.
684 696 Indeed, some players quickly understood all the rules of the game and how to maximize their score,
685 697 while others struggled to make points during the whole session, even with our help.
... ... @@ -701,7 +713,8 @@
701 713 Moreover, the top five sessions in terms of percentage solved (all sessions with more than 65\%) come from the four different game conditions.
702 714  
703 715 We used linear regression to test if the percentage of the problem solved is, to some extent, directly proportional to the total experience points accumulated
704   -by all the players during a session (graph not shown). The linear function obtained had a coefficient of correlation $r = 0.89$ and a coefficient of determination
  716 +by all the players during a session, which is a good way to measure the skills of the players during each session.
  717 +The linear function obtained (graph not shown) had a coefficient of correlation $r = 0.89$ and a coefficient of determination
705 718 $r^2=0.79$, which shows a certain level of correlation. The different game conditions are obviously creating some of the observed variance.
706 719 %Notice that a game session with many good players combining for a high total of experience points does not guarantee that
707 720 %a bigger percentage of the solution will be found by the players.
... ... @@ -717,7 +730,7 @@
717 730 Based on the questionnaire filled by the players before playing the game, and the global leaderboard of all the players from all the sessions put together,
718 731 we tried to find similarities between the top players. Table~\ref{tab_playerStats} shows the most interesting differences between the top 12 players
719 732 and the rest of the players. In the questionnaires, players had to indicate their age category (between 21 and 25 for example), their own evaluation
720   -of their puzzle solving abilities and a range of hours of time spent playing video games every week.
  733 +of their puzzle solving abilities and a range of hours spent playing video games every week.
721 734  
722 735 The average age of the two groups of players
723 736 was calculated by taking the middle point of the age categories. The average age of the top 12 players was about $2.5$ years younger than the one of
... ... @@ -727,7 +740,6 @@
727 740 rest of the players.
728 741  
729 742 \begin{table}[h]
730   -\caption{Average statistics on the top 12 players vs the others}\label{tab_playerStats}
731 743 \begin{center}
732 744 \begin{tabular}{ccc}\hline
733 745 & Top 12 players & Others\\
... ... @@ -736,6 +748,7 @@
736 748 Game time & 10.00 & 4.11\\\hline
737 749 \end{tabular}
738 750 \end{center}
  751 +\caption{Average statistics on the top 12 players vs the others}\label{tab_playerStats}
739 752 \end{table}
740 753  
741 754 \section{Conclusion}
... ... @@ -743,7 +756,7 @@
743 756 We implemented a human computing game that uses a market, skills and challenges in order to solve a problem collaboratively. The problem that is solved
744 757 by the players in our game is a graph problem that can be easily translated into a color matching game. The total number of colors used in the tests was small
745 758 enough so that we were able to compute an exact solution and evaluate the performance of the players. We organized 12 game sessions of 10 players with
746   -four different game conditions (3 times each).
  759 +four different game conditions (three times each).
747 760  
748 761 Our tests showed without a doubt that the market is a useful tool to help players build longer solutions (sequences, in our case). In addition,
749 762 it also makes the game a lot more dynamic and players mentioned that they really enjoyed this aspect of the game.
750 763  
751 764  
752 765  
... ... @@ -757,20 +770,21 @@
757 770 in order to be effective, the difficulty needs to be well-balanced. Challenges that are too easy ({\em Sell/buy challenge} for example) or
758 771 too hard ({\em Specific colors in common challenge} for example) do not affect the game significantly.
759 772  
760   -Although the great variability in the participants' skills made it very difficult to make direct comparisons between the different game conditions
  773 +Although the great variability in the participants' personal skills made it very difficult to make direct comparisons between the different game conditions
761 774 in regards to the percentage of the solutions found, we showed that the percentage solved is to a certain extent proportional to the total experience gained
762   -by all players during a game session.
  775 +by all players during a game session. Therefore, the percentage of the problem solved is clearly not only dependent on the features present in the game, but
  776 +also on the participants' ability to be good at the game.
763 777  
764 778 Finally, it seems that younger players who play video games on a regular basis and
765 779 have a strong self evaluation of their puzzle solving skills are able to understand the rules
766 780 of the game and find winning strategies faster than the average participant.
767 781  
768   -\section{Acknowledgments}
  782 +%\section{Acknowledgments}
769 783  
770   -First and foremost, the authors wish to thank all the players who made this study possible.
771   -The authors would also like to thank Jean-Fran\c{c}ois Bourbeau, Mathieu Blanchette, Derek Ruths and Edward Newell for their help with the initial design of the game,
772   -and Alexandre Leblanc for his helpful advice on the statistical tests.
773   -Finally, the authors wish to thank Silvia Juliana Leon Mantilla and Shu Hayakawa for their help with the organization of the game sessions and the recruitment of participants.
  784 +%First and foremost, the authors wish to thank all the players who made this study possible.
  785 +%The authors would also like to thank Jean-Fran\c{c}ois Bourbeau, Mathieu Blanchette, Derek Ruths and Edward Newell for their help with the initial design of the game,
  786 +%and Alexandre Leblanc for his helpful advice on the statistical tests.
  787 +%Finally, the authors wish to thank Silvia Juliana Leon Mantilla and Shu Hayakawa for their help with the organization of the game sessions and the recruitment of participants.
774 788  
775 789 % REFERENCES FORMAT
776 790 % References must be the same font size as other body text.