Commit a0ca0a31b87814028d62901a60aaa34f91ee5f31

1 parent
d64f6c27e8

Exists in
master

### Updated Results: section Testing hypothesis 2 (skills)

Showing
**7 changed files**
with
**97 additions**
and
**104 deletions**
Side-by-side Diff

CHIpaper/Figs/boxplot_BK.pdf
View file @
a0ca0a3

CHIpaper/Figs/boxplot_CE.pdf
View file @
a0ca0a3

CHIpaper/Figs/boxplot_MT.pdf
View file @
a0ca0a3

CHIpaper/Figs/boxplot_SC_nbCols.pdf
View file @
a0ca0a3

CHIpaper/Figs/boxplot_SC_seqLength.pdf
View file @
a0ca0a3

CHIpaper/MarketPaper.pdf
View file @
a0ca0a3

CHIpaper/MarketPaper.tex
View file @
a0ca0a3

... | ... | @@ -419,13 +419,13 @@ |

419 | 419 | lengths for all the sequences sold to the system during a game session do not follow a normal distribution, we used a non-parametric test (Kruskal-Wallis) to |

420 | 420 | verify if the sequence lengths of the different game sessions seem to come from the same distributions. |

421 | 421 | The Kruskal-Wallis test revealed a significant effect of the game conditions on the sequence lengths without considering super circles |

422 | -(${\chi}^2(2) = 1391.7$, $p < \num{2.2e-16}$) and also when considering super circles (${\chi}^2(2) = 1388.4$, $p < \num{2.2e-16}$). | |

422 | +(${\chi}^2(2) = 1391.7$, $p < 2.2E-16$) and also when considering super circles (${\chi}^2(2) = 1388.4$, $p < 2.2E-16$). | |

423 | 423 | |

424 | 424 | We then made a post hoc test (Dunn's test) to do pairwise comparisons between all the groups. With or without considering super circles, all the game conditions |

425 | 425 | were shown to be significantly different ($p < 0.01$), except a few shown in table~\ref{tab_Dunn}. Note that the strongest similarities are found between |

426 | 426 | the three 'All' groups and the three 'No market' groups. Some of the 'No skills' experiments are found to be similar to the 'All' groups, which could indicate |

427 | 427 | that the presence of the skills have a very limited effect on the sequence length. The NC experiment is found to be similar to two 'No market' groups, but that |

428 | -can be explained by the fact the players for the NC experiment were very weak (see section~\ref{sect_hyp4}). | |

428 | +can be explained by the fact the players for the NC experiment were very weak (see Section Testing hypothesis 4). | |

429 | 429 | |

430 | 430 | \begin{table}[h] |

431 | 431 | \caption{Similar groups of sequence length distributions, as reported by Dunn's test. An 'n' in the table represent a similar pair when not considering |

432 | 432 | |

433 | 433 | |

434 | 434 | |

435 | 435 | |

436 | 436 | |

437 | 437 | |

438 | 438 | |

439 | 439 | |

440 | 440 | |

441 | 441 | |

442 | 442 | |

443 | 443 | |

444 | 444 | |

445 | 445 | |

446 | 446 | |

447 | 447 | |

448 | 448 | |

449 | 449 | |

450 | 450 | |

451 | 451 | |

452 | 452 | |

... | ... | @@ -444,141 +444,134 @@ |

444 | 444 | \end{table} |

445 | 445 | |

446 | 446 | %WILL HAVE TO MOVE THE FOLLOWING SENTENCES TO HYPOTHESIS 4 SECTION |

447 | -Notice that even in | |

448 | -the two sessions for which we had the smallest total experience (see Figure~\ref{fig_totalXP}), both averages of sequence lengths were larger than the averages | |

449 | -of the game session without the market. Those observations confirm that the market is helping the players in the creation of longer sequences. | |

447 | +%Notice that even in | |

448 | +%the two sessions for which we had the smallest total experience (see Figure~\ref{fig_totalXP}), both averages of sequence lengths were larger than the averages | |

449 | +%of the game session without the market. Those observations confirm that the market is helping the players in the creation of longer sequences. | |

450 | 450 | |

451 | 451 | \subsection{Testing hypothesis 2: the benefits of a skill system} |

452 | 452 | |

453 | -We implemented the skill system for two reasons: (1) to encourage the players to level-up, because the reward is a skill point, and (2) to influence indirectly | |

453 | +We implemented the skill system for two reasons: (1) to give the players more incentive to accumulate experience points as fast as possible, because | |

454 | +the reward for leveling-up is an additional skill point, and (2) to influence indirectly | |

454 | 455 | the players into doing actions that are either improving the solutions collected by the system or helpful to the other players (which in the end will also |

455 | 456 | improve the solutions). In our game, two skills were related to the market ({\em Buyout King} and {\em Master Trader}) and two skills were related to building |

456 | 457 | sequences ({\em Color Expert} and {\em Sequence Collector}). In the following paragraphs, we will analyze how those four skills affected the strategies and actions |

457 | -of the players. | |

458 | +of the players. Note that when some players lost all their money in the game, they had to start a new game. In our results, we count both | |

459 | +new games as if they were played by different players, since the players who restart might choose a different set of skills the second time. That explains | |

460 | +why the total number of players is larger than 120. Players of the 'No skills' game condition were considered and put automatically in the without skill | |

461 | +group. | |

458 | 462 | |

459 | 463 | \subsubsection{Buyout King} |

460 | 464 | |

461 | 465 | The {\em Buyout King} skill allows the players to reduce the price of buying a sequence from another player (which we call a buyout). The idea behind this skill |

462 | -was to encourage the players to buy small sequences from others so that they can improve it before selling them to the system. In other words, it's the | |

463 | -action of buying a partial solution in order to improve it. | |

466 | +was to encourage the players to buy small sequences built by other players so that they could improve them before selling them back to the system. | |

467 | +In other words, a buyout is the action of buying a partial solution made by another player in order to improve it. | |

464 | 468 | |

465 | -\begin{table}[h] | |

466 | -\caption{Statitics on the number of buyouts by players with and without the {\em Buyout King} skill}\label{tab_buyoutKing} | |

467 | -\begin{center} | |

468 | -\begin{tabular}{cccccc}\hline | |

469 | - & Nb Players & Min & Max & Median & Average\\ | |

470 | -With & 18 & 0 & 70 & 10 & 20.06\\ | |

471 | -Without & 30 & 0 & 61 & 0.50 & 10.40\\\hline | |

472 | -\end{tabular} | |

473 | -\end{center} | |

474 | -\end{table} | |

469 | +\begin{figure}[htbp] | |

470 | + \begin{center} | |

471 | + \includegraphics[width=\halfWidth]{Figs/boxplot_BK.pdf} | |

472 | + \vspace{0cm} | |

473 | + \caption{Boxplot of the number of buyouts made by players with (37 players) and without (66 players) the {\em Buyout King} skill. | |

474 | + }\label{fig_boxplotBK} | |

475 | + \end{center} | |

476 | +\end{figure} | |

475 | 477 | |

476 | -Table~\ref{tab_buyoutKing} shows statitics for the players who have put at least one skill point in the {\em Buyout King} skill and the players who did not | |

477 | -use the skill at all. We were interested in the number of buyouts that the players with the skill were doing compared to the rest of the players. Note that | |

478 | -since this skill is related to the market, we did not consider the 'No market' session for these results. Also notice that the total number of players is | |

479 | -larger than 40 (4 sessions of 10 participants) because when the players went bankrupt (not enough money to do anything), we allowed them to restart the game | |

480 | -with a new username. | |

478 | +Figure~\ref{fig_boxplotBK} shows statitics for the players who have put at least one skill point in the {\em Buyout King} skill and the players who did not | |

479 | +use the skill at all. We were interested in the number of buyouts that the players with the skill were making compared to the rest of the players. Note that | |

480 | +since this skill is related to the market, we did not consider the 'No market' sessions for these results. | |

481 | 481 | |

482 | -On average, players who spent a skill point in the {\em Buyout King} skill bought sequences twice as much as the other players. Moreover, the median value for | |

483 | -the players without the skill ($0.5$, which is the mean of the two middle values 0 and 1) shows that half of the players without the skill did not use the | |

484 | -buyout at all. | |

482 | +The median value for players who spent a skill point in the {\em Buyout King} skill is 15, while the median value for | |

483 | +the players without the skill is 1.5, indicating that half of the players without the skill did not use the | |

484 | +buyout at all or used it only once. Since the distribution of the number of buyouts is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis | |

485 | +with $p = 2.6E-10$), we used a Mann-Whitney's U test to compare the medians of the two groups. We found a significant effect of the presence | |

486 | +of this skill on the medians ($U = 1629.5$, $p = 0.004$, effect size $r = 0.28$). | |

485 | 487 | |

486 | 488 | \subsubsection{Master Trader} |

487 | 489 | |

488 | -The {\em Master Trader} skill allows the players to get bonus money on top of the regular market price for each individual circle. This skill was put | |

489 | -in the game in order to boost the activity on the market by encouraging more players to send the circles that they don't need to players who need it the most. | |

490 | +The {\em Master Trader} skill allows the players to get bonus money in addition to the regular market price for each circle they sell individually. This skill was put | |

491 | +in the game in order to increase the activity on the market by encouraging more players to send the circles that they don't need to players who need it the most. | |

490 | 492 | |

491 | -\begin{table}[h] | |

492 | -\caption{Statitics on the number circles sold by players with and without the {\em Master Trader} skill}\label{tab_masterTrader} | |

493 | -\begin{center} | |

494 | -\begin{tabular}{cccccc}\hline | |

495 | - & Nb Players & Min & Max & Median & Average\\ | |

496 | -With & 14 & 0 & 288 & 61 & 76\\ | |

497 | -Without & 34 & 0 & 139 & 8 & 22.74\\\hline | |

498 | -\end{tabular} | |

499 | -\end{center} | |

500 | -\end{table} | |

493 | +\begin{figure}[htbp] | |

494 | + \begin{center} | |

495 | + \includegraphics[width=\halfWidth]{Figs/boxplot_MT.pdf} | |

496 | + \vspace{0cm} | |

497 | + \caption{Boxplot of the number of circles sold individually by players with (33 players) and without (70 players) the {\em Master Trader} skill. | |

498 | + }\label{fig_boxplotMT} | |

499 | + \end{center} | |

500 | +\end{figure} | |

501 | 501 | |

502 | -Table~\ref{tab_masterTrader} shows statistics for the players who have put at least one skill point in the {\em Master Trader} skill and all the other players. | |

502 | +Figure~\ref{fig_boxplotMT} shows statistics for the players who have put at least one skill point in the {\em Master Trader} skill and all the other players. | |

503 | 503 | We were interested in comparing the number of individual circles that were sold to another player for the two different categories. Once again, since this |

504 | -skill depends on the presence of the market, we did not consider the 'No market' experiment in the results shown. | |

504 | +skill depends on the presence of the market, we did not consider the 'No market' experiments in the results shown. | |

505 | 505 | |

506 | -On average, the players who had selected the {\em Master Trader} skill sold more than three times more circles on the market than the rest of the players. | |

507 | -One player in particular took full advantage of the skill and sold an impressive total of 288 circles to other players during a game session. | |

506 | +The median value for the players who had selected the {\em Master Trader} skill (73) is more than three times larger than the one for the rest of the players (21.5). | |

507 | +Since the distribution of the number of circles sold individually is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis | |

508 | +with $p = 5.3E-13$), we used a Mann-Whitney's U test to compare the medians of the two groups. We found a significant effect of the presence | |

509 | +of the {\em Master Trader} skill on the medians ($U = 1633.5$, $p = 7.2E-4$, effect size $r = 0.33$). | |

508 | 510 | |

509 | 511 | \subsubsection{Color Expert} |

510 | 512 | |

511 | -The {\em Color Expert} skill gives a bonus multiplier to the scoring function of sequences with more than one color in common. This skill was implemented in | |

512 | -order to give extra motivation to build sequences with many colors in common, since they are harder to build. Indeed, more concentration is needed to identify | |

513 | -circles with more than one color in common. | |

513 | +The {\em Color Expert} skill gives a bonus multiplier to the scoring function for sequences with more than one color in common. This skill was implemented in | |

514 | +order to give extra motivation to build sequences with many colors in common, since they are harder to build. Indeed, more focus is needed from the player to match | |

515 | +many cicles with more than one color in common. | |

514 | 516 | |

515 | -\begin{table}[h] | |

516 | -\caption{Statitics on the percentage of sequences with more than one color in common sold by players with and without the {\em Color Expert} skill}\label{tab_colorExpert} | |

517 | -\begin{center} | |

518 | -\begin{tabular}{cccccc}\hline | |

519 | - & Nb Players & Min & Max & Median & Average\\ | |

520 | -With & 37 & 0\% & 100\% & 26.32\% & 29.76\%\\ | |

521 | -Without & 25 & 0\% & 97.30\% & 30\% & 34.40\%\\\hline | |

522 | -\end{tabular} | |

523 | -\end{center} | |

524 | -\end{table} | |

517 | +\begin{figure}[htbp] | |

518 | + \begin{center} | |

519 | + \includegraphics[width=\halfWidth]{Figs/boxplot_CE.pdf} | |

520 | + \vspace{0cm} | |

521 | + \caption{Boxplot of the proportion of sequences with more than one color in common sold by players with (94 players) and without (49 players) the {\em Color Expert} skill. | |

522 | + }\label{fig_boxplotCE} | |

523 | + \end{center} | |

524 | +\end{figure} | |

525 | 525 | |

526 | -In table~\ref{tab_colorExpert}, we compare the statistical values related to the percentage of multicolored sequences sold by players with the {\em Color Expert} skill | |

527 | -and players without it. Interestingly, the players without the skill, on average, were selling a higher proportion of sequences with more than one color in common. This | |

528 | -can be explained by the fact that one of the main goals of the game is to create sequences with as many colors in common as possible, whether the player selects this skill or not. | |

526 | +In Figure~\ref{fig_boxplotCE}, we show the comparison of the proportion of multicolored sequences sold by players with the {\em Color Expert} skill | |

527 | +and players without it. Interestingly, the median values for both groups are almost identical: 0.317 (or 31.7\%) for the players with the skill and | |

528 | +0.313 (or 31.3\%) for the players without the skill. The distribution of the proportion of multicolored sequences was not normal (the Shapiro-Wilk | |

529 | +test rejected the null hypothesis with $p = 0.3E-4$), so we did a Mann-Whithney's U test to compare the medians. As expected, the test failed to reject | |

530 | +the null hypothesis that the values were sampled from the same distribution ($p = 0.89$). | |

529 | 531 | |

530 | -Table~\ref{tab_colorExpert2} shows the same statistics, but for a different grouping of the players. In order to eliminate the noise that could come from players | |

531 | -who select the skill by accident, players with at least two skill points in the skill were compared to all the others. With this grouping, we can see that all the | |

532 | -players who put two points or more in the skill at least built $6.25\%$ of their sequences with more than one color. On average, the players with more than 2 points | |

533 | -in the {\em Color Expert} skill had a higher percentage of multicolored sequences than the rest of the players ($35.80\%$ vs $29.17\%$). | |

532 | +We conclude that the {\em Color Expert} skill does not affect the behavior of the players. This can be explained by the fact that one of the main goals of | |

533 | +the game is to create sequences with as many colors in common as possible, whether the player selects this skill or not. | |

534 | 534 | |

535 | -\begin{table}[h] | |

536 | -\caption{Statitics on the percentage of sequences with more than one color in common sold by players with at least two | |

537 | -points in the {\em Color Expert} skill vs the others}\label{tab_colorExpert2} | |

538 | -\begin{center} | |

539 | -\begin{tabular}{cccccc}\hline | |

540 | - & Nb Players & Min & Max & Median & Average\\ | |

541 | -$\geq$ 2 pts & 23 & 6.25\% & 100\% & 31.40\% & 35.80\%\\ | |

542 | -Others & 39 & 0\% & 97.30\% & 27.87\% & 29.17\%\\\hline | |

543 | -\end{tabular} | |

544 | -\end{center} | |

545 | -\end{table} | |

546 | - | |

547 | 535 | \subsubsection{Sequence Collector} |

548 | 536 | |

549 | -Finally, every point in the {\em Sequence Collector} skill gives an additionnal slot to build a sequence. Because of the limited size of the player's hand and the | |

550 | -limited number of sequence slots, it's hard to build long sequences. This is why we added the {\em Sequence Collector} skill in the game. | |

537 | +Every point in the {\em Sequence Collector} skill gives an additionnal slot to build a sequence. Because of the limited size of the player's hand and the | |

538 | +limited number of sequence slots, it's hard to build long sequences with many colors in common. It is for both the sequence length and | |

539 | +the number of colors in common that we added the {\em Sequence Collector} skill in the game. | |

551 | 540 | |

552 | -\begin{table}[h] | |

553 | -\caption{Statistics on the average sequence length of players with and without the {\em Sequence Collector} skill}\label{tab_seqCollector} | |

554 | -\begin{center} | |

555 | -\begin{tabular}{cccccc}\hline | |

556 | - & Nb Players & Min & Max & Median & Average\\ | |

557 | -With & 22 & 2.33 & 9 & 5.04 & 5.04\\ | |

558 | -Without & 40 & 2.39 & 9.05 & 5.40 & 5.31\\\hline | |

559 | -\end{tabular} | |

560 | -\end{center} | |

561 | -\end{table} | |

541 | +\begin{figure}[htbp] | |

542 | + \begin{center} | |

543 | + \includegraphics[width=\halfWidth]{Figs/boxplot_SC_seqLength.pdf} | |

544 | + \vspace{0cm} | |

545 | + \caption{Boxplot of the average sequence length of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill. | |

546 | + }\label{fig_boxplotSC_seqLength} | |

547 | + \end{center} | |

548 | +\end{figure} | |

562 | 549 | |

563 | -Table~\ref{tab_seqCollector} shows statistics on the average sequence length of the players with the {\em Sequence Collector} skill and the other players. | |

564 | -We can observe here a pattern that is similar to the one of the {\em Color Expert} skill: players without the skill are, on average, selling sequences that | |

565 | -are longer. Once again, this can be explained by the fact that selling long sequences is one of the two main goals of the game, and is one of the main components | |

550 | +We first compared the average sequence length of sequences built by players with the {\em Sequence Collector} skill and the ones built by the rest of | |

551 | +players~\ref{fig_boxplotSC_seqLength}. While the median value for the players without the skill (5.63) is a little bit larger than the one for the players | |

552 | +with the skill (5.12), the averages of both groups are actually similar (5.61 and 5.56 in the same order). Since the distribution of the average | |

553 | +sequence lengths were not normal (the Shapiro-Wilk test rejected the null hypothesis with $p = 0.0057$), we did a Mann Whitney's U test to compare the | |

554 | +medians of both groups. The test failed to reject the null hypothesis that the values were sampled from the same distribution ($p = 0.69$). Thus, | |

555 | +there is no evidence that the {\em Sequence Collector} skill helps players build longer sequences. This tends to confirm what we mentioned earlier (in Section | |

556 | +Testing hypothesis 1): the presence of the skills in general does not seem to affect the length of the sequences built by players. | |

557 | +Once again, this can be explained by the fact that selling long sequences is one of the two main goals of the game, and is one of the main components | |

566 | 558 | of the scoring function. |

567 | 559 | |

568 | -Once again, we made the comparison between the players with at least two points in the {\em Sequence Collector} skill and all the others. Just like with the | |

569 | -{\em Color Expert} skill, we observe that the players who really choose to specialize in this ability are doing better than the other players. As shown in | |

570 | -Table~\ref{tab_seqCollector2}, the average sequence length is a little bit higher for players who put more than one skill point in {\em Sequence Collector}. | |

560 | +\begin{figure}[htbp] | |

561 | + \begin{center} | |

562 | + \includegraphics[width=\halfWidth]{Figs/boxplot_SC_nbCols.pdf} | |

563 | + \vspace{0cm} | |

564 | + \caption{Boxplot of the average number of colors in common of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill. | |

565 | + }\label{fig_boxplotSC_nbCols} | |

566 | + \end{center} | |

567 | +\end{figure} | |

571 | 568 | |

572 | -\begin{table}[h] | |

573 | -\caption{Statistics on the average sequence length of players with at least two points in the {\em Sequence Collector} skill vs the others}\label{tab_seqCollector2} | |

574 | -\begin{center} | |

575 | -\begin{tabular}{cccccc}\hline | |

576 | - & Nb Players & Min & Max & Median & Average\\ | |

577 | -$\geq$ 2 pts & 10 & 2.78 & 9 & 5.10 & 5.35\\ | |

578 | -Others & 52 & 2.33 & 9.05 & 5.11 & 5.19\\\hline | |

579 | -\end{tabular} | |

580 | -\end{center} | |

581 | -\end{table} | |

569 | +We then compared the average number of colors in common of the sequences built by players with and without the {\em Sequence collector} skill~\ref{fig_boxplotSC_nbCols}. | |

570 | +The median value for the players without the skill (1.58) is 14\% lower than the one for the players with the skill (1.80). Since the distribution of the average | |

571 | +number of colors in common is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis | |

572 | +with $p = 1.2E-7$), we used a Mann-Whitney's U test to compare the medians of the two groups and we found a significant effect of the presence | |

573 | +of this skill on the medians ($U = 3113$, $p = 0.01$, effect size $r = 0.21$). The {\em Sequence collector} skill is thus helping players | |

574 | +to build sequences with more colors, by allowing them to store unfinished sequences of multiple colors in the additional slots until they are able to complete them. | |

582 | 575 | |

583 | 576 | \subsection{Testing hypothesis 3: the usefulness of the challenge system} |

584 | 577 | |

... | ... | @@ -666,7 +659,7 @@ |

666 | 659 | doing actions that are not specific to a certain subset of colors. Even if the market should be helpful in finding circles with the required |

667 | 660 | subset of colors, it seems highly probable that the players felt that this type of challenge was too hard and never tried to complete it. |

668 | 661 | |

669 | -\subsection{Testing hypothesis 4: relationship between total experience and percentage solved}\label{sect_hyp4} | |

662 | +\subsection{Testing hypothesis 4: relationship between total experience and percentage solved} | |

670 | 663 | %Coming back on the 4 tests, total game xp vs percentage of problem solved |

671 | 664 | As mentioned in the Experiments section, the initial plan was to measure the impact of each feature by analyzing how much of the problem can be solved |

672 | 665 | by the players in each of the game sessions. Interestingly, we observed a larger than expected variance in the participants' skills which made it practically |