Commit 4adfe24734b2531b2e35e19fb83d54716d56a135

1 parent
544aed6779

Exists in
master

### Final touches. Small tweaks all over the paper. This is the version that was submitted to CHI2016.

Showing
**2 changed files**
with
**75 additions**
and
**61 deletions**
Side-by-side Diff

CHIpaper/MarketPaper.pdf
View file @
4adfe24

CHIpaper/MarketPaper.tex
View file @
4adfe24

... | ... | @@ -103,7 +103,7 @@ |

103 | 103 | % End of preamble. Here it comes the document. |

104 | 104 | \begin{document} |

105 | 105 | |

106 | -\title{Collaborative solving in a human computing game using a market, skills and challenges} | |

106 | +\title{Collaborative Solving in a Human Computing Game Using a Market, Skills and Challenges} | |

107 | 107 | |

108 | 108 | \numberofauthors{3} |

109 | 109 | \author{% |

... | ... | @@ -125,8 +125,8 @@ |

125 | 125 | |

126 | 126 | \begin{abstract} |

127 | 127 | Using a human computing game to solve a problem that has a large search space is not straightforward. The difficulty of using such an approach |

128 | -comes from the following facts: (1) it would be overwhelming for a single player to show him or her the complete search space and at the same time, | |

129 | -(2) it is impossible to find an optimal solution without considering all the available data. In this paper, we present a human computing | |

128 | +comes from the following facts: (i) it would be overwhelming for a single player to show him or her the complete search space and at the same time, | |

129 | +(ii) it is impossible to find an optimal solution without considering all the available data. In this paper, we present a human computing | |

130 | 130 | game that uses a market, skills and a challenge system to help the players solve a graph problem in a collaborative manner. The results obtained |

131 | 131 | during 12 game sessions of 10 players show that the market helps players to build larger solutions. We also show that a skill system and, to a lesser extent, a |

132 | 132 | challenge system can be used to influence and guide the players towards producing better solutions. |

133 | 133 | |

134 | 134 | |

135 | 135 | |

... | ... | @@ -135,17 +135,17 @@ |

135 | 135 | \keywords{Human computing; Collaboration; Crowdsourcing; Graph problem; Game; Market; Trading game; Skills; Challenges.} |

136 | 136 | |

137 | 137 | \section{Introduction} |

138 | -Human-computation and crowd-sourcing are now perceived as valuable techniques to help solving difficult computational problems. In order to make the best use of human skills in these systems, it is important to be able to characterize the expertise and performance of humans as individual and even more importantly as groups. | |

138 | +Human-computation and crowd-sourcing are now perceived as valuable techniques to help solving difficult computational problems. In order to make the best use of human skills in these systems, it is important to be able to characterize the expertise and performance of humans as individuals and even more importantly as groups. | |

139 | 139 | |

140 | -Currently, popular crowd-computing platform such as Amazon Mechanical Turk (AMT) \cite{Buhrmester01012011, Paolacci} or Crowdcrafting \cite{Crowdcrafting} are based on similars divide-and-conquer architectures, where the initial problem is decomposed into smaller sub-tasks that are distributed to individual workers and then aggregated to build a solution. In particular, these systems prevent any interaction between workers in order to prevent groupthink phenomena and bias in the solution \cite{Lorenz:2011aa}. | |

140 | +Currently, popular crowd-computing platforms such as Amazon Mechanical Turk (AMT) \cite{Buhrmester01012011, Paolacci} or Crowdcrafting \cite{Crowdcrafting} are based on similar divide-and-conquer architectures, where the initial problem is decomposed into smaller sub-tasks that are distributed to individual workers and then aggregated to build a solution. In particular, these systems prevent any interaction between workers in order to prevent groupthink phenomena and bias in the solution \cite{Lorenz:2011aa}. | |

141 | 141 | |

142 | 142 | However, such constraints are necessarily limiting the capacity of the system to harness the cognitive power of crowds and make full benefit of collective intelligence. For instance, iterative combinations of crowdsourced contributions can help enhancing creativity \cite{DBLP:conf/chi/YuN11}. The usefulness of parallelizing workflows has also been suggested for tasks accepting broad varieties of answers \cite{DBLP:conf/chi/Little10}. |

143 | 143 | |

144 | 144 | The benefits of developing recommendation systems or coordination methods in collaborative environments has been demonstrated \cite{DBLP:conf/cscw/KitturK08,DBLP:conf/cscw/DowKKH12,DBLP:conf/chi/ZhangLMGPH12}. Therefore, in order to gain expressivity and improve their performance, the next generation of human-computation systems will certainly need to implement mechanisms to promote and control the collaboration between workers. Nonetheless, before transitioning to this model, it is important to first estimate the potential gains in productivity, and quantify the usefulness of the mechanisms and incentives to promote collaborative solving and prevent groupthink. |

145 | 145 | |

146 | -Historically, computation on graphs has proven to be a good model to study the performance of humans in solving complex combinatorial problems \cite{Kearns:2006aa}. Experiments have been conducted to evaluate the dynamics of crowds collaborating at solving graph problems \cite{DBLP:journals/cacm/Kearns12} but still, little is known about the efficiency of various modes of interaction. | |

146 | +Historically, computation on graphs has proven to be a good model to study the performance of humans in solving complex combinatorial problems \cite{Kearns:2006aa}. Experiments have been conducted to evaluate the dynamics of crowds collaborating at solving graph problems \cite{DBLP:journals/cacm/Kearns12} but still, little is known about the efficiency of the various modes of interaction. | |

147 | 147 | |

148 | -In this paper, we propose a formal framework to study human collaborative solving. We embed a combinatorial graph problem into a novel multiplayer game-with-a-purpose \cite{DBLP:conf/chi/AhnD04,DBLP:conf/aaai/HoCH07}, which will be used engage participants and analyze collective performances. More precisely, we design a market game in which players can sell and buy their solutions, and coupled this platform with (i) a skills systems to enhance the efficiency of specific gaming strategies and (ii) a challenge systems to guide the work of the crowd. We use this game to investigate the validity of the following hypotheses. | |

148 | +In this paper, we propose a formal framework to study human collaborative solving. We embed a combinatorial graph problem into a novel multiplayer game-with-a-purpose \cite{DBLP:conf/chi/AhnD04,DBLP:conf/aaai/HoCH07}, which will be used to engage participants and analyze collective performances. More precisely, we design a market game in which players can sell and buy solutions or bits of information, and couple this platform with (i) a skills system to enhance the efficiency of specific gaming strategies and (ii) a challenge systems to guide the work of the crowd. We use this game to investigate the validity of the following hypotheses. | |

149 | 149 | |

150 | 150 | \subsection{Hypotheses} |

151 | 151 | |

152 | 152 | |

... | ... | @@ -155,10 +155,12 @@ |

155 | 155 | \item A market system will help the players build better solutions. |

156 | 156 | \item A skill system is useful to orient the players into doing specific actions that are beneficial to the game and other players. |

157 | 157 | \item A challenge system is effective in encouraging the players to do a specific action in the game. |

158 | - \item The collected solutions are better when all the 3 features are present in a game session, independently of the players' skills. | |

158 | + \item The collected solutions are better when all the three features are present in a game session, independently of the players' personal skills. | |

159 | 159 | \end{enumerate} |

160 | 160 | |

161 | -To answer these questions, we conducted a study on 120 participants using different variants of our market game. Our results confirm the benefits of using a trading platform to produce better solutions. Interestingly, we also found that a skills system helps to promote actions favorable to the collective solving process (e.g. increasing the diversity of intermediate solutions), but that the efficiency of a skill is reduced if it is designed to solve one of the primary objectives of the game. Finally, we observed that a precise parametrization of challenges (i.e. finding an appropriate difficulty, nor too easy, nor too difficult) is required to result in an improvement of the quality of the collective work. | |

161 | +To answer these questions, we conducted a study on 120 participants using different variants of our market game. Our results confirm the benefits of using a trading platform to produce better solutions. Interestingly, we also found that a skills system helps to promote actions that are favorable to the collective solving process, | |

162 | +%(e.g. increasing the diversity of intermediate solutions) | |

163 | +but that the efficiency of a skill is reduced if it is designed to help solve one of the primary objectives of the game. Finally, we observed that a precise parametrization of challenges (i.e. finding an appropriate difficulty, nor too easy, nor too difficult) is required to result in an improvement of the quality of the collective work. | |

162 | 164 | |

163 | 165 | Our game is freely available at \texttt{URL:TBA}, and can be used as a platform for further independent studies. |

164 | 166 | |

165 | 167 | |

... | ... | @@ -179,13 +181,19 @@ |

179 | 181 | colors of the vertices, it is possible to show the players only the colored vertices. To solve the problem, the players have to find the largests sets |

180 | 182 | of circles with colors in common, for all possible subsets of colors. |

181 | 183 | |

184 | +Note that it is not our goal to compare the performance of players with the performance of computers in solving this problem. With a limited number of colors | |

185 | +(like six in our tests), the exact algorithm can solve the problem in seconds. For this study, we required a | |

186 | +problem that was structured enough so that we could | |

187 | +easily calculate the optimal solution and evaluate the performance of the players depending on what features were on or off and also the effect of the different | |

188 | +features on the quality of solutions. | |

182 | 189 | |

183 | 190 | |

184 | 191 | \section{Presentation of the game} |

185 | 192 | |

186 | 193 | \subsection{Goal of the game} |

187 | 194 | |

188 | -The main objective of the game is to build {\em sequences} ({\em i.e.} sets) of circles that are as long as possible and contain as many colors in common | |

195 | +The main objective of the game is to build {\em sequences} ({\em i.e.} sets) of circles (circles represent vertices of the graph) that (i) are as long as possible | |

196 | +and (ii) contain as many colors in common | |

189 | 197 | as possible. Circles used by the players to build the sequences either come random packages bought from the system or they come from another player |

190 | 198 | through the market. The sequences can then be sold to the system for a certain amount of game money, which is determined by a scoring function that takes into |

191 | 199 | account the length and the number of colors in common of the sequence. |

192 | 200 | |

... | ... | @@ -198,10 +206,10 @@ |

198 | 206 | that is proportional to the difficulty of building the sequence. More precisely, |

199 | 207 | we calculated the average length $L_n$ of all solutions for each $n$ number of colors. The base score is simply the reciprocal of this average ($1/L_n$) multiplied |

200 | 208 | by a balancing factor (505 in our case). The balancing factor was chosen in order to get a score of 500 for a sequence of length 10 with only one color in common, |

201 | -which is exactly the price of two random packages of circles. | |

209 | +which is exactly the price of two random packages of circles. Also notice that the value of a sequence is exponential in relation to its length, which is | |

210 | +to encourage players to build the longest possible sequences. | |

202 | 211 | |

203 | 212 | \begin{table}[h] |

204 | -\caption{Value of the base score depending on the number of colors in common}\label{tab_baseScore} | |

205 | 213 | \begin{center} |

206 | 214 | \begin{tabular}{cc}\hline |

207 | 215 | Number of colors & Base score\\ |

... | ... | @@ -214,6 +222,7 @@ |

214 | 222 | 6 & 72\\\hline |

215 | 223 | \end{tabular} |

216 | 224 | \end{center} |

225 | +\caption{Value of the base score depending on the number of colors in common}\label{tab_baseScore} | |

217 | 226 | \end{table} |

218 | 227 | |

219 | 228 | \subsection{Game interface} |

220 | 229 | |

221 | 230 | |

222 | 231 | |

... | ... | @@ -221,21 +230,21 @@ |

221 | 230 | The game client and the server were built in Java 1.7. |

222 | 231 | As shown in Figure~\ref{fig_interface}, the game interface can be divided into 3 parts: the player information panel, the game panel and the market panel. |

223 | 232 | |

224 | -\subsubsection{Player information panel} | |

233 | +\subsubsection{A: Player information panel} | |

225 | 234 | |

226 | 235 | This panel simply contains information on the player's wallet, the current level of the player and has three buttons, allowing the player to open |

227 | 236 | dialogs showing information on the current challenge, the skills (see section Skills for a description of the available skills) and the leaderboard. |

228 | 237 | One experience point is given to the player for each game dollar that he/she wins. The player can lose game money, but cannot lose experience points |

229 | -(experience points can only go up). | |

238 | +(experience points can only go up). Every time a player levels-up, he/she gets a skill point that can be used to improve any of the skills. | |

230 | 239 | |

231 | -\subsubsection{Game panel} | |

240 | +\subsubsection{B: Game panel} | |

232 | 241 | |

233 | 242 | The first component of the game panel is the 'My sequence' panel, which shows the current sequence that is being built by the player. The maximum size |

234 | 243 | of a sequence is 10. Colors in common in the |

235 | 244 | sequence are indicated by a thick black border surrounding the colors in the circles. Players can use the arrows to switch between the different sequence slots |

236 | 245 | (2 sequence slots are available at the start of the game). The current value of the sequence is shown at the right, and the price for adding one more circle |

237 | 246 | with the same colors in common is shown right below in gray. Finally, the sell button allows the player to sell the current sequence to the system: the sequence then |

238 | -disappears and the money is given to the player. | |

247 | +disappears and the money is given to the player. Selling a sequence is equivalent to submitting a solution to the system. | |

239 | 248 | |

240 | 249 | The second component is the 'My hand' panel, which can contain up to 20 circles. Players can add a circle to the sequence by clicking on it. Circles are |

241 | 250 | represented by their colors and by a price label (in a black box). The price corresponds to the current value of the circle on the market. Clicking on |

... | ... | @@ -248,7 +257,7 @@ |

248 | 257 | Finally, the bottom panel is a news feed, showing information on the game state, like the remaining time to complete the challenge and the last transactions |

249 | 258 | completed by the player for example. |

250 | 259 | |

251 | -\subsubsection{Market panel} | |

260 | +\subsubsection{C: Market panel} | |

252 | 261 | |

253 | 262 | At the top of the market panel, buttons allow the player to create bids for circles or to buy random packages (or bags) of circles. |

254 | 263 | The 'Random bag' costs \$250 and contains 5 circles with fewer colors. The 'Premium bag' costs \$500 and contains 5 circles |

255 | 264 | |

... | ... | @@ -256,11 +265,13 @@ |

256 | 265 | |

257 | 266 | Right below the buttons is the 'Automatic bids' panel, which allows the player to get automatic bids for circles corresponding to the sequences that |

258 | 267 | he or she is building. A percentage of profit for the price of the automatic bids can be set with the slider. |

259 | -The profit is defined as the money the player would make by adding one more SNP with the same colors in the current sequence (difference between the gray and black prices). | |

268 | +The profit is defined as the money the player would make by adding one more circle with the same colors in the current sequence (difference between the gray and black prices | |

269 | +above the Sell button). | |

260 | 270 | |

261 | 271 | The 'My bids' panel shows all the bids that the player currently has on the market. The bid price is shown below the circle (in the black box). On the right side |

262 | 272 | of the circle is the number of sequences with the same colors that the player can buy from other players (in the blue box). |

263 | -Clicking on the blue box opens a window showing the list of sequences that can be bought. Buying a sequence from another player is called a 'buyout'. | |

273 | +Clicking on the blue box opens a window showing the list of sequences that can be bought. Buying a sequence from another player is called a 'buyout' (see | |

274 | +the following subsection for a more detailed description of buyouts). | |

264 | 275 | |

265 | 276 | Finally, the last panel at the bottom shows the last circle or sequence that was bought by the player. |

266 | 277 | |

... | ... | @@ -276,8 +287,8 @@ |

276 | 287 | |

277 | 288 | \subsection{Market} |

278 | 289 | |

279 | -The market has three functions: allow the players to exchange circles through a bidding system, allow players to buy sequences built and sold by other | |

280 | -players so that they can be improved, and merge together sequences of length 10 to create super circles that are then put back in the game. | |

290 | +The market has three functions: (i) allow the players to exchange circles through a bidding system, (ii) allow players to buy sequences built and sold by other | |

291 | +players so that they can be improved, and (iii) merge together sequences of length 10 to create super circles that are then put back in the game. | |

281 | 292 | |

282 | 293 | For every subset of colors, the server has a list of all the bids that are currently on the market. The value of |

283 | 294 | the highest bid on the market is shown below every circle under the possession of the players. When a circle is sold by a player, it is sent through |

... | ... | @@ -289,7 +300,7 @@ |

289 | 300 | |

290 | 301 | Finally, the game system creates a super circle every time a sequence of length 10 is sold by a player. A super circle of level 2 (representing 10 circles) |

291 | 302 | counts as two circles when put in a sequence. Super circles can be of any level (a sequence of 10 super circles of level 2 form a super circle of level 3, and so on). |

292 | -The objective is to remove the limitation of the maximum sequence size imposed by the game interface. | |

303 | +The idea behind the creation of the super circles was to remove the limitation of the maximum sequence size imposed by the game interface. | |

293 | 304 | |

294 | 305 | \subsection{Skills} |

295 | 306 | |

... | ... | @@ -298,10 +309,10 @@ |

298 | 309 | guide the player in doing actions that are beneficial to the system or to the other players: |

299 | 310 | |

300 | 311 | \begin{itemize} |

301 | -\item {\em Buyout King}: lowers the price of buying a sequence from another player; | |

302 | -\item {\em Color Expert}: gives a bonus to selling sequences that have more than one color in common; | |

303 | -\item {\em Sequence Collector}: gives an additional sequence slot; | |

304 | -\item {\em Master Trader}: gives a bonus to selling SNPs to other players. | |

312 | +\item {\em Buyout King}: lowers the price of buying a sequence from another player (goal: encourage buyouts); | |

313 | +\item {\em Color Expert}: gives a bonus to selling sequences that have more than one color in common (goal: push players to build more multicolored sequences); | |

314 | +\item {\em Sequence Collector}: gives an additional sequence slot (goal: give more space to encourage the creation of longer sequences with more colors in common); | |

315 | +\item {\em Master Trader}: gives a bonus to selling circles to other players (goal: promote the selling of individual circles). | |

305 | 316 | \end{itemize} |

306 | 317 | |

307 | 318 | \subsection{Challenge system} |

... | ... | @@ -319,7 +330,7 @@ |

319 | 330 | |

320 | 331 | Basically, the system continuously monitors the activities of the players and decreases or increases the probabilities of each challenge type. |

321 | 332 | The next challenge is then selected using a multinomial sampling on these probabilities. The number of times $T$ that the challenge-related action must be |

322 | -completed is selected randomly between 2 and 4. The prize that is awarded for completing the challenge is equal to $1500 * T$. | |

333 | +completed is selected randomly between 2 and 4. The prize that is awarded for completing the challenge is equal to \$$1500 * T$. | |

323 | 334 | |

324 | 335 | \section{Experiments} |

325 | 336 | |

... | ... | @@ -391,7 +402,7 @@ |

391 | 402 | |

392 | 403 | \subsection{Testing hypothesis 1: the efficiency of the market} |

393 | 404 | |

394 | -The market system we implemented in the game allows the players to exchange circles and partial solutions (sequences). The main goal of the market | |

405 | +The market system we implemented in the game allows the players to exchange circles and partial solutions (in the form of buyouts). The main goal of the market | |

395 | 406 | is to help the players in building longer sequences. |

396 | 407 | |

397 | 408 | %\begin{tabular}{ccc}\hline |

398 | 409 | |

399 | 410 | |

... | ... | @@ -430,11 +441,11 @@ |

430 | 441 | were shown to be significantly different ($p < 0.01$), except a few shown in table~\ref{tab_Dunn}. Note that the strongest similarities are found between |

431 | 442 | the three 'All' groups and the three 'No market' groups. Some of the 'No skills' experiments are found to be similar to the 'All' groups, which could indicate |

432 | 443 | that the presence of the skills have a very limited effect on the sequence length. The NC experiment is found to be similar to two 'No market' groups, but that |

433 | -can be explained by the fact the players for the NC experiment were very weak (see Section Testing hypothesis 4). | |

444 | +can be explained by the fact the players for the NC experiment were very weak (as can be seen by the total experience gained during that session in | |

445 | +Figure~\ref{fig_totalXP}). | |

434 | 446 | |

447 | +\setlength{\tabcolsep}{4pt} | |

435 | 448 | \begin{table}[h] |

436 | - \caption{Similar groups of sequence length distributions, as reported by Dunn's test. An 'n' in the table represent a similar pair when not considering | |

437 | - super circles, and an 's' in the table represents a similar pair when considering super circles.}\label{tab_Dunn} | |

438 | 449 | \begin{center} |

439 | 450 | \begin{tabular}{c|cccccccc}\hline |

440 | 451 | & A-2 & A-3 & NS & NS-2 & NS-3 & NM & NM-2 & NM-3\\\hline |

... | ... | @@ -446,6 +457,8 @@ |

446 | 457 | NM & & & & & & & n/s & n \\\hline |

447 | 458 | \end{tabular} |

448 | 459 | \end{center} |

460 | +\caption{Similar groups of sequence length distributions, as reported by Dunn's test. An 'n' in the table represents a similar pair when not considering | |

461 | + super circles, and an 's' in the table represents a similar pair when considering super circles.}\label{tab_Dunn} | |

449 | 462 | \end{table} |

450 | 463 | |

451 | 464 | %WILL HAVE TO MOVE THE FOLLOWING SENTENCES TO HYPOTHESIS 4 SECTION |

... | ... | @@ -455,8 +468,8 @@ |

455 | 468 | |

456 | 469 | \subsection{Testing hypothesis 2: the benefits of a skill system} |

457 | 470 | |

458 | -We implemented the skill system for two reasons: (1) to give the players more incentive to accumulate experience points as fast as possible, because | |

459 | -the reward for leveling-up is an additional skill point, and (2) to influence indirectly | |

471 | +We implemented the skill system for two reasons: (i) to give the players more incentive to accumulate experience points as fast as possible, because | |

472 | +the reward for leveling-up is an additional skill point, and (ii) to influence indirectly | |

460 | 473 | the players into doing actions that are either improving the solutions collected by the system or helpful to the other players (which in the end will also |

461 | 474 | improve the solutions). In our game, two skills were related to the market ({\em Buyout King} and {\em Master Trader}) and two skills were related to building |

462 | 475 | sequences ({\em Color Expert} and {\em Sequence Collector}). In the following paragraphs, we will analyze how those four skills affected the strategies and actions |

... | ... | @@ -473,7 +486,7 @@ |

473 | 486 | |

474 | 487 | \begin{figure}[htbp] |

475 | 488 | \begin{center} |

476 | - \includegraphics[width=\halfWidth]{Figs/boxplot_BK.pdf} | |

489 | + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_BK.pdf} | |

477 | 490 | \vspace{0cm} |

478 | 491 | \caption{Boxplot of the number of buyouts made by players with (37 players) and without (66 players) the {\em Buyout King} skill. |

479 | 492 | }\label{fig_boxplotBK} |

... | ... | @@ -497,7 +510,7 @@ |

497 | 510 | |

498 | 511 | \begin{figure}[htbp] |

499 | 512 | \begin{center} |

500 | - \includegraphics[width=\halfWidth]{Figs/boxplot_MT.pdf} | |

513 | + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_MT.pdf} | |

501 | 514 | \vspace{0cm} |

502 | 515 | \caption{Boxplot of the number of circles sold individually by players with (33 players) and without (70 players) the {\em Master Trader} skill. |

503 | 516 | }\label{fig_boxplotMT} |

504 | 517 | |

... | ... | @@ -517,11 +530,11 @@ |

517 | 530 | |

518 | 531 | The {\em Color Expert} skill gives a bonus multiplier to the scoring function for sequences with more than one color in common. This skill was implemented in |

519 | 532 | order to give extra motivation to build sequences with many colors in common, since they are harder to build. Indeed, more focus is needed from the player to match |

520 | -many cicles with more than one color in common. | |

533 | +many circles with more than one color in common. | |

521 | 534 | |

522 | 535 | \begin{figure}[htbp] |

523 | 536 | \begin{center} |

524 | - \includegraphics[width=\halfWidth]{Figs/boxplot_CE.pdf} | |

537 | + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_CE.pdf} | |

525 | 538 | \vspace{0cm} |

526 | 539 | \caption{Boxplot of the proportion of sequences with more than one color in common sold by players with (94 players) and without (49 players) the {\em Color Expert} skill. |

527 | 540 | }\label{fig_boxplotCE} |

... | ... | @@ -545,7 +558,7 @@ |

545 | 558 | |

546 | 559 | \begin{figure}[htbp] |

547 | 560 | \begin{center} |

548 | - \includegraphics[width=\halfWidth]{Figs/boxplot_SC_seqLength.pdf} | |

561 | + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_SC_seqLength.pdf} | |

549 | 562 | \vspace{0cm} |

550 | 563 | \caption{Boxplot of the average sequence length of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill. |

551 | 564 | }\label{fig_boxplotSC_seqLength} |

... | ... | @@ -553,7 +566,7 @@ |

553 | 566 | \end{figure} |

554 | 567 | |

555 | 568 | We first compared the average sequence length of sequences built by players with the {\em Sequence Collector} skill and the ones built by the rest of |

556 | -players~\ref{fig_boxplotSC_seqLength}. While the median value for the players without the skill (5.63) is a little bit larger than the one for the players | |

569 | +players (see Figure~\ref{fig_boxplotSC_seqLength}). While the median value for the players without the skill (5.63) is a little bit larger than the one for the players | |

557 | 570 | with the skill (5.12), the averages of both groups are actually similar (5.61 and 5.56 in the same order). Since the distribution of the average |

558 | 571 | sequence lengths were not normal (the Shapiro-Wilk test rejected the null hypothesis with $p = 0.0057$), we did a Mann Whitney's U test to compare the |

559 | 572 | medians of both groups. The test failed to reject the null hypothesis that the values were sampled from the same distribution ($p = 0.69$). Thus, |

560 | 573 | |

... | ... | @@ -564,15 +577,15 @@ |

564 | 577 | |

565 | 578 | \begin{figure}[htbp] |

566 | 579 | \begin{center} |

567 | - \includegraphics[width=\halfWidth]{Figs/boxplot_SC_nbCols.pdf} | |

580 | + \includegraphics[width=\halfWidth-1in]{Figs/boxplot_SC_nbCols.pdf} | |

568 | 581 | \vspace{0cm} |

569 | 582 | \caption{Boxplot of the average number of colors in common of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill. |

570 | 583 | }\label{fig_boxplotSC_nbCols} |

571 | 584 | \end{center} |

572 | 585 | \end{figure} |

573 | 586 | |

574 | -We then compared the average number of colors in common of the sequences built by players with and without the {\em Sequence collector} skill~\ref{fig_boxplotSC_nbCols}. | |

575 | -The median value for the players without the skill (1.58) is 14\% lower than the one for the players with the skill (1.80). Since the distribution of the average | |

587 | +We then compared the average number of colors in common of the sequences built by players with and without the {\em Sequence collector} skill (see Figure~\ref{fig_boxplotSC_nbCols}). | |

588 | +The median value for the players without the skill (1.58) is 12\% lower than the one for the players with the skill (1.80). Since the distribution of the average | |

576 | 589 | number of colors in common is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis |

577 | 590 | with $p = 1.2E-7$), we used a Mann-Whitney's U test to compare the medians of the two groups and we found a significant effect of the presence |

578 | 591 | of this skill on the medians ($U = 3113$, $p = 0.01$, effect size $r = 0.21$). The {\em Sequence collector} skill is thus helping players |

... | ... | @@ -600,7 +613,7 @@ |

600 | 613 | \vspace{0cm} |

601 | 614 | \caption{Average number of colors in the sequences with and without the {\em Minimum number of colors challenge} active. 'A', 'A-2' and 'A-3' |

602 | 615 | represent the tests with all the features present, 'NS', 'NS-2' and 'NS-3' represent the tests without the skills, and 'NM', 'NM-2' and 'NM-3' represents |

603 | - the test without the market. | |

616 | + the tests without the market. | |

604 | 617 | }\label{fig_minNbCols} |

605 | 618 | \end{center} |

606 | 619 | \end{figure} |

607 | 620 | |

608 | 621 | |

... | ... | @@ -620,14 +633,14 @@ |

620 | 633 | \begin{center} |

621 | 634 | \includegraphics[width=\halfWidth]{Figs/minSeqLength.pdf} |

622 | 635 | \vspace{0cm} |

623 | - \caption{Average sequence length with and without the {\em Minimum sequence length challenge active}. 'A', 'A-2' and 'A-3' | |

636 | + \caption{Average sequence length with and without the {\em Minimum sequence length challenge} active. 'A', 'A-2' and 'A-3' | |

624 | 637 | represent the tests with all the features present, 'NS', 'NS-2' and 'NS-3' represent the tests without the skills, and 'NM', 'NM-2' and 'NM-3' represents |

625 | - the test without the market. | |

638 | + the tests without the market. | |

626 | 639 | }\label{fig_minSeqLength} |

627 | 640 | \end{center} |

628 | 641 | \end{figure} |

629 | 642 | |

630 | -The means of the average sequence lengths during the challenge and for the rest of the time are 5.38 and 5.08 respectively. Since the distribution | |

643 | +The means of all the average sequence lengths during the challenge and for the rest of the time are 5.38 and 5.08 respectively. Since the distribution | |

631 | 644 | of the averages of sequence lengths is normal (Shapiro-Wilk $p = 0.27$), we used a Welch's t-test to compare those means, but the test wasn't able |

632 | 645 | to prove that those means are significantly different ($t(16)=0.79$, $p = 0.44$). |

633 | 646 | |

... | ... | @@ -645,9 +658,8 @@ |

645 | 658 | \begin{center} |

646 | 659 | \includegraphics[width=\halfWidth]{Figs/sellBuySNP.pdf} |

647 | 660 | \vspace{0cm} |

648 | - \caption{Number of individual circles sold on the market per minute with and without the Sell/buy challenge active. 'A', 'A-2' and 'A-3' | |

649 | - represent the tests with all the features present, 'NS', 'NS-2' and 'NS-3' represent the tests without the skills, and 'NM', 'NM-2' and 'NM-3' represents | |

650 | - the test without the market. | |

661 | + \caption{Number of individual circles sold on the market per minute with and without the {\em Sell/buy challenge} active. 'A', 'A-2' and 'A-3' | |

662 | + represent the tests with all the features present, and 'NS', 'NS-2' and 'NS-3' represent the tests without the skills. | |

651 | 663 | }\label{fig_sellBuySNP} |

652 | 664 | \end{center} |

653 | 665 | \end{figure} |

... | ... | @@ -679,7 +691,7 @@ |

679 | 691 | by the players in each of the game sessions. Our initial hypothesis was that players who have access to all the game features should |

680 | 692 | be able to solve more of the problem. |

681 | 693 | |

682 | -Interestingly, we observed a larger than expected variance in the participants' skills which made it sometimes | |

694 | +Interestingly, we observed a larger than expected variance in the participants' personal skills which made it sometimes | |

683 | 695 | difficult to compare one game session with another in terms of the percentage of the problem that was solved. |

684 | 696 | Indeed, some players quickly understood all the rules of the game and how to maximize their score, |

685 | 697 | while others struggled to make points during the whole session, even with our help. |

... | ... | @@ -701,7 +713,8 @@ |

701 | 713 | Moreover, the top five sessions in terms of percentage solved (all sessions with more than 65\%) come from the four different game conditions. |

702 | 714 | |

703 | 715 | We used linear regression to test if the percentage of the problem solved is, to some extent, directly proportional to the total experience points accumulated |

704 | -by all the players during a session (graph not shown). The linear function obtained had a coefficient of correlation $r = 0.89$ and a coefficient of determination | |

716 | +by all the players during a session, which is a good way to measure the skills of the players during each session. | |

717 | +The linear function obtained (graph not shown) had a coefficient of correlation $r = 0.89$ and a coefficient of determination | |

705 | 718 | $r^2=0.79$, which shows a certain level of correlation. The different game conditions are obviously creating some of the observed variance. |

706 | 719 | %Notice that a game session with many good players combining for a high total of experience points does not guarantee that |

707 | 720 | %a bigger percentage of the solution will be found by the players. |

... | ... | @@ -717,7 +730,7 @@ |

717 | 730 | Based on the questionnaire filled by the players before playing the game, and the global leaderboard of all the players from all the sessions put together, |

718 | 731 | we tried to find similarities between the top players. Table~\ref{tab_playerStats} shows the most interesting differences between the top 12 players |

719 | 732 | and the rest of the players. In the questionnaires, players had to indicate their age category (between 21 and 25 for example), their own evaluation |

720 | -of their puzzle solving abilities and a range of hours of time spent playing video games every week. | |

733 | +of their puzzle solving abilities and a range of hours spent playing video games every week. | |

721 | 734 | |

722 | 735 | The average age of the two groups of players |

723 | 736 | was calculated by taking the middle point of the age categories. The average age of the top 12 players was about $2.5$ years younger than the one of |

... | ... | @@ -727,7 +740,6 @@ |

727 | 740 | rest of the players. |

728 | 741 | |

729 | 742 | \begin{table}[h] |

730 | -\caption{Average statistics on the top 12 players vs the others}\label{tab_playerStats} | |

731 | 743 | \begin{center} |

732 | 744 | \begin{tabular}{ccc}\hline |

733 | 745 | & Top 12 players & Others\\ |

... | ... | @@ -736,6 +748,7 @@ |

736 | 748 | Game time & 10.00 & 4.11\\\hline |

737 | 749 | \end{tabular} |

738 | 750 | \end{center} |

751 | +\caption{Average statistics on the top 12 players vs the others}\label{tab_playerStats} | |

739 | 752 | \end{table} |

740 | 753 | |

741 | 754 | \section{Conclusion} |

... | ... | @@ -743,7 +756,7 @@ |

743 | 756 | We implemented a human computing game that uses a market, skills and challenges in order to solve a problem collaboratively. The problem that is solved |

744 | 757 | by the players in our game is a graph problem that can be easily translated into a color matching game. The total number of colors used in the tests was small |

745 | 758 | enough so that we were able to compute an exact solution and evaluate the performance of the players. We organized 12 game sessions of 10 players with |

746 | -four different game conditions (3 times each). | |

759 | +four different game conditions (three times each). | |

747 | 760 | |

748 | 761 | Our tests showed without a doubt that the market is a useful tool to help players build longer solutions (sequences, in our case). In addition, |

749 | 762 | it also makes the game a lot more dynamic and players mentioned that they really enjoyed this aspect of the game. |

750 | 763 | |

751 | 764 | |

752 | 765 | |

... | ... | @@ -757,20 +770,21 @@ |

757 | 770 | in order to be effective, the difficulty needs to be well-balanced. Challenges that are too easy ({\em Sell/buy challenge} for example) or |

758 | 771 | too hard ({\em Specific colors in common challenge} for example) do not affect the game significantly. |

759 | 772 | |

760 | -Although the great variability in the participants' skills made it very difficult to make direct comparisons between the different game conditions | |

773 | +Although the great variability in the participants' personal skills made it very difficult to make direct comparisons between the different game conditions | |

761 | 774 | in regards to the percentage of the solutions found, we showed that the percentage solved is to a certain extent proportional to the total experience gained |

762 | -by all players during a game session. | |

775 | +by all players during a game session. Therefore, the percentage of the problem solved is clearly not only dependent on the features present in the game, but | |

776 | +also on the participants' ability to be good at the game. | |

763 | 777 | |

764 | 778 | Finally, it seems that younger players who play video games on a regular basis and |

765 | 779 | have a strong self evaluation of their puzzle solving skills are able to understand the rules |

766 | 780 | of the game and find winning strategies faster than the average participant. |

767 | 781 | |

768 | -\section{Acknowledgments} | |

782 | +%\section{Acknowledgments} | |

769 | 783 | |

770 | -First and foremost, the authors wish to thank all the players who made this study possible. | |

771 | -The authors would also like to thank Jean-Fran\c{c}ois Bourbeau, Mathieu Blanchette, Derek Ruths and Edward Newell for their help with the initial design of the game, | |

772 | -and Alexandre Leblanc for his helpful advice on the statistical tests. | |

773 | -Finally, the authors wish to thank Silvia Juliana Leon Mantilla and Shu Hayakawa for their help with the organization of the game sessions and the recruitment of participants. | |

784 | +%First and foremost, the authors wish to thank all the players who made this study possible. | |

785 | +%The authors would also like to thank Jean-Fran\c{c}ois Bourbeau, Mathieu Blanchette, Derek Ruths and Edward Newell for their help with the initial design of the game, | |

786 | +%and Alexandre Leblanc for his helpful advice on the statistical tests. | |

787 | +%Finally, the authors wish to thank Silvia Juliana Leon Mantilla and Shu Hayakawa for their help with the organization of the game sessions and the recruitment of participants. | |

774 | 788 | |

775 | 789 | % REFERENCES FORMAT |

776 | 790 | % References must be the same font size as other body text. |