Commit bce8d6a9f0965180a0fafc26a0f01f7ffe094f81

Authored by waldispuhl
Exists in master

Merge branch 'master' of jwgitlab.cs.mcgill.ca:jerome/market-game

None?

Showing 7 changed files Side-by-side Diff

CHIpaper/Figs/boxplot_BK.pdf View file @ bce8d6a

No preview for this file type

CHIpaper/Figs/boxplot_CE.pdf View file @ bce8d6a

No preview for this file type

CHIpaper/Figs/boxplot_MT.pdf View file @ bce8d6a

No preview for this file type

CHIpaper/Figs/boxplot_SC_nbCols.pdf View file @ bce8d6a

No preview for this file type

CHIpaper/Figs/boxplot_SC_seqLength.pdf View file @ bce8d6a

No preview for this file type

CHIpaper/MarketPaper.pdf View file @ bce8d6a

No preview for this file type

CHIpaper/MarketPaper.tex View file @ bce8d6a
... ... @@ -419,13 +419,13 @@
419 419 lengths for all the sequences sold to the system during a game session do not follow a normal distribution, we used a non-parametric test (Kruskal-Wallis) to
420 420 verify if the sequence lengths of the different game sessions seem to come from the same distributions.
421 421 The Kruskal-Wallis test revealed a significant effect of the game conditions on the sequence lengths without considering super circles
422   -(${\chi}^2(2) = 1391.7$, $p < \num{2.2e-16}$) and also when considering super circles (${\chi}^2(2) = 1388.4$, $p < \num{2.2e-16}$).
  422 +(${\chi}^2(2) = 1391.7$, $p < 2.2E-16$) and also when considering super circles (${\chi}^2(2) = 1388.4$, $p < 2.2E-16$).
423 423  
424 424 We then made a post hoc test (Dunn's test) to do pairwise comparisons between all the groups. With or without considering super circles, all the game conditions
425 425 were shown to be significantly different ($p < 0.01$), except a few shown in table~\ref{tab_Dunn}. Note that the strongest similarities are found between
426 426 the three 'All' groups and the three 'No market' groups. Some of the 'No skills' experiments are found to be similar to the 'All' groups, which could indicate
427 427 that the presence of the skills have a very limited effect on the sequence length. The NC experiment is found to be similar to two 'No market' groups, but that
428   -can be explained by the fact the players for the NC experiment were very weak (see section~\ref{sect_hyp4}).
  428 +can be explained by the fact the players for the NC experiment were very weak (see Section Testing hypothesis 4).
429 429  
430 430 \begin{table}[h]
431 431 \caption{Similar groups of sequence length distributions, as reported by Dunn's test. An 'n' in the table represent a similar pair when not considering
432 432  
433 433  
434 434  
435 435  
436 436  
437 437  
438 438  
439 439  
440 440  
441 441  
442 442  
443 443  
444 444  
445 445  
446 446  
447 447  
448 448  
449 449  
450 450  
451 451  
452 452  
... ... @@ -444,141 +444,134 @@
444 444 \end{table}
445 445  
446 446 %WILL HAVE TO MOVE THE FOLLOWING SENTENCES TO HYPOTHESIS 4 SECTION
447   -Notice that even in
448   -the two sessions for which we had the smallest total experience (see Figure~\ref{fig_totalXP}), both averages of sequence lengths were larger than the averages
449   -of the game session without the market. Those observations confirm that the market is helping the players in the creation of longer sequences.
  447 +%Notice that even in
  448 +%the two sessions for which we had the smallest total experience (see Figure~\ref{fig_totalXP}), both averages of sequence lengths were larger than the averages
  449 +%of the game session without the market. Those observations confirm that the market is helping the players in the creation of longer sequences.
450 450  
451 451 \subsection{Testing hypothesis 2: the benefits of a skill system}
452 452  
453   -We implemented the skill system for two reasons: (1) to encourage the players to level-up, because the reward is a skill point, and (2) to influence indirectly
  453 +We implemented the skill system for two reasons: (1) to give the players more incentive to accumulate experience points as fast as possible, because
  454 +the reward for leveling-up is an additional skill point, and (2) to influence indirectly
454 455 the players into doing actions that are either improving the solutions collected by the system or helpful to the other players (which in the end will also
455 456 improve the solutions). In our game, two skills were related to the market ({\em Buyout King} and {\em Master Trader}) and two skills were related to building
456 457 sequences ({\em Color Expert} and {\em Sequence Collector}). In the following paragraphs, we will analyze how those four skills affected the strategies and actions
457   -of the players.
  458 +of the players. Note that when some players lost all their money in the game, they had to start a new game. In our results, we count both
  459 +new games as if they were played by different players, since the players who restart might choose a different set of skills the second time. That explains
  460 +why the total number of players is larger than 120. Players of the 'No skills' game condition were considered and put automatically in the without skill
  461 +group.
458 462  
459 463 \subsubsection{Buyout King}
460 464  
461 465 The {\em Buyout King} skill allows the players to reduce the price of buying a sequence from another player (which we call a buyout). The idea behind this skill
462   -was to encourage the players to buy small sequences from others so that they can improve it before selling them to the system. In other words, it's the
463   -action of buying a partial solution in order to improve it.
  466 +was to encourage the players to buy small sequences built by other players so that they could improve them before selling them back to the system.
  467 +In other words, a buyout is the action of buying a partial solution made by another player in order to improve it.
464 468  
465   -\begin{table}[h]
466   -\caption{Statitics on the number of buyouts by players with and without the {\em Buyout King} skill}\label{tab_buyoutKing}
467   -\begin{center}
468   -\begin{tabular}{cccccc}\hline
469   - & Nb Players & Min & Max & Median & Average\\
470   -With & 18 & 0 & 70 & 10 & 20.06\\
471   -Without & 30 & 0 & 61 & 0.50 & 10.40\\\hline
472   -\end{tabular}
473   -\end{center}
474   -\end{table}
  469 +\begin{figure}[htbp]
  470 + \begin{center}
  471 + \includegraphics[width=\halfWidth]{Figs/boxplot_BK.pdf}
  472 + \vspace{0cm}
  473 + \caption{Boxplot of the number of buyouts made by players with (37 players) and without (66 players) the {\em Buyout King} skill.
  474 + }\label{fig_boxplotBK}
  475 + \end{center}
  476 +\end{figure}
475 477  
476   -Table~\ref{tab_buyoutKing} shows statitics for the players who have put at least one skill point in the {\em Buyout King} skill and the players who did not
477   -use the skill at all. We were interested in the number of buyouts that the players with the skill were doing compared to the rest of the players. Note that
478   -since this skill is related to the market, we did not consider the 'No market' session for these results. Also notice that the total number of players is
479   -larger than 40 (4 sessions of 10 participants) because when the players went bankrupt (not enough money to do anything), we allowed them to restart the game
480   -with a new username.
  478 +Figure~\ref{fig_boxplotBK} shows statitics for the players who have put at least one skill point in the {\em Buyout King} skill and the players who did not
  479 +use the skill at all. We were interested in the number of buyouts that the players with the skill were making compared to the rest of the players. Note that
  480 +since this skill is related to the market, we did not consider the 'No market' sessions for these results.
481 481  
482   -On average, players who spent a skill point in the {\em Buyout King} skill bought sequences twice as much as the other players. Moreover, the median value for
483   -the players without the skill ($0.5$, which is the mean of the two middle values 0 and 1) shows that half of the players without the skill did not use the
484   -buyout at all.
  482 +The median value for players who spent a skill point in the {\em Buyout King} skill is 15, while the median value for
  483 +the players without the skill is 1.5, indicating that half of the players without the skill did not use the
  484 +buyout at all or used it only once. Since the distribution of the number of buyouts is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis
  485 +with $p = 2.6E-10$), we used a Mann-Whitney's U test to compare the medians of the two groups. We found a significant effect of the presence
  486 +of this skill on the medians ($U = 1629.5$, $p = 0.004$, effect size $r = 0.28$).
485 487  
486 488 \subsubsection{Master Trader}
487 489  
488   -The {\em Master Trader} skill allows the players to get bonus money on top of the regular market price for each individual circle. This skill was put
489   -in the game in order to boost the activity on the market by encouraging more players to send the circles that they don't need to players who need it the most.
  490 +The {\em Master Trader} skill allows the players to get bonus money in addition to the regular market price for each circle they sell individually. This skill was put
  491 +in the game in order to increase the activity on the market by encouraging more players to send the circles that they don't need to players who need it the most.
490 492  
491   -\begin{table}[h]
492   -\caption{Statitics on the number circles sold by players with and without the {\em Master Trader} skill}\label{tab_masterTrader}
493   -\begin{center}
494   -\begin{tabular}{cccccc}\hline
495   - & Nb Players & Min & Max & Median & Average\\
496   -With & 14 & 0 & 288 & 61 & 76\\
497   -Without & 34 & 0 & 139 & 8 & 22.74\\\hline
498   -\end{tabular}
499   -\end{center}
500   -\end{table}
  493 +\begin{figure}[htbp]
  494 + \begin{center}
  495 + \includegraphics[width=\halfWidth]{Figs/boxplot_MT.pdf}
  496 + \vspace{0cm}
  497 + \caption{Boxplot of the number of circles sold individually by players with (33 players) and without (70 players) the {\em Master Trader} skill.
  498 + }\label{fig_boxplotMT}
  499 + \end{center}
  500 +\end{figure}
501 501  
502   -Table~\ref{tab_masterTrader} shows statistics for the players who have put at least one skill point in the {\em Master Trader} skill and all the other players.
  502 +Figure~\ref{fig_boxplotMT} shows statistics for the players who have put at least one skill point in the {\em Master Trader} skill and all the other players.
503 503 We were interested in comparing the number of individual circles that were sold to another player for the two different categories. Once again, since this
504   -skill depends on the presence of the market, we did not consider the 'No market' experiment in the results shown.
  504 +skill depends on the presence of the market, we did not consider the 'No market' experiments in the results shown.
505 505  
506   -On average, the players who had selected the {\em Master Trader} skill sold more than three times more circles on the market than the rest of the players.
507   -One player in particular took full advantage of the skill and sold an impressive total of 288 circles to other players during a game session.
  506 +The median value for the players who had selected the {\em Master Trader} skill (73) is more than three times larger than the one for the rest of the players (21.5).
  507 +Since the distribution of the number of circles sold individually is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis
  508 +with $p = 5.3E-13$), we used a Mann-Whitney's U test to compare the medians of the two groups. We found a significant effect of the presence
  509 +of the {\em Master Trader} skill on the medians ($U = 1633.5$, $p = 7.2E-4$, effect size $r = 0.33$).
508 510  
509 511 \subsubsection{Color Expert}
510 512  
511   -The {\em Color Expert} skill gives a bonus multiplier to the scoring function of sequences with more than one color in common. This skill was implemented in
512   -order to give extra motivation to build sequences with many colors in common, since they are harder to build. Indeed, more concentration is needed to identify
513   -circles with more than one color in common.
  513 +The {\em Color Expert} skill gives a bonus multiplier to the scoring function for sequences with more than one color in common. This skill was implemented in
  514 +order to give extra motivation to build sequences with many colors in common, since they are harder to build. Indeed, more focus is needed from the player to match
  515 +many cicles with more than one color in common.
514 516  
515   -\begin{table}[h]
516   -\caption{Statitics on the percentage of sequences with more than one color in common sold by players with and without the {\em Color Expert} skill}\label{tab_colorExpert}
517   -\begin{center}
518   -\begin{tabular}{cccccc}\hline
519   - & Nb Players & Min & Max & Median & Average\\
520   -With & 37 & 0\% & 100\% & 26.32\% & 29.76\%\\
521   -Without & 25 & 0\% & 97.30\% & 30\% & 34.40\%\\\hline
522   -\end{tabular}
523   -\end{center}
524   -\end{table}
  517 +\begin{figure}[htbp]
  518 + \begin{center}
  519 + \includegraphics[width=\halfWidth]{Figs/boxplot_CE.pdf}
  520 + \vspace{0cm}
  521 + \caption{Boxplot of the proportion of sequences with more than one color in common sold by players with (94 players) and without (49 players) the {\em Color Expert} skill.
  522 + }\label{fig_boxplotCE}
  523 + \end{center}
  524 +\end{figure}
525 525  
526   -In table~\ref{tab_colorExpert}, we compare the statistical values related to the percentage of multicolored sequences sold by players with the {\em Color Expert} skill
527   -and players without it. Interestingly, the players without the skill, on average, were selling a higher proportion of sequences with more than one color in common. This
528   -can be explained by the fact that one of the main goals of the game is to create sequences with as many colors in common as possible, whether the player selects this skill or not.
  526 +In Figure~\ref{fig_boxplotCE}, we show the comparison of the proportion of multicolored sequences sold by players with the {\em Color Expert} skill
  527 +and players without it. Interestingly, the median values for both groups are almost identical: 0.317 (or 31.7\%) for the players with the skill and
  528 +0.313 (or 31.3\%) for the players without the skill. The distribution of the proportion of multicolored sequences was not normal (the Shapiro-Wilk
  529 +test rejected the null hypothesis with $p = 0.3E-4$), so we did a Mann-Whithney's U test to compare the medians. As expected, the test failed to reject
  530 +the null hypothesis that the values were sampled from the same distribution ($p = 0.89$).
529 531  
530   -Table~\ref{tab_colorExpert2} shows the same statistics, but for a different grouping of the players. In order to eliminate the noise that could come from players
531   -who select the skill by accident, players with at least two skill points in the skill were compared to all the others. With this grouping, we can see that all the
532   -players who put two points or more in the skill at least built $6.25\%$ of their sequences with more than one color. On average, the players with more than 2 points
533   -in the {\em Color Expert} skill had a higher percentage of multicolored sequences than the rest of the players ($35.80\%$ vs $29.17\%$).
  532 +We conclude that the {\em Color Expert} skill does not affect the behavior of the players. This can be explained by the fact that one of the main goals of
  533 +the game is to create sequences with as many colors in common as possible, whether the player selects this skill or not.
534 534  
535   -\begin{table}[h]
536   -\caption{Statitics on the percentage of sequences with more than one color in common sold by players with at least two
537   -points in the {\em Color Expert} skill vs the others}\label{tab_colorExpert2}
538   -\begin{center}
539   -\begin{tabular}{cccccc}\hline
540   - & Nb Players & Min & Max & Median & Average\\
541   -$\geq$ 2 pts & 23 & 6.25\% & 100\% & 31.40\% & 35.80\%\\
542   -Others & 39 & 0\% & 97.30\% & 27.87\% & 29.17\%\\\hline
543   -\end{tabular}
544   -\end{center}
545   -\end{table}
546   -
547 535 \subsubsection{Sequence Collector}
548 536  
549   -Finally, every point in the {\em Sequence Collector} skill gives an additionnal slot to build a sequence. Because of the limited size of the player's hand and the
550   -limited number of sequence slots, it's hard to build long sequences. This is why we added the {\em Sequence Collector} skill in the game.
  537 +Every point in the {\em Sequence Collector} skill gives an additionnal slot to build a sequence. Because of the limited size of the player's hand and the
  538 +limited number of sequence slots, it's hard to build long sequences with many colors in common. It is for both the sequence length and
  539 +the number of colors in common that we added the {\em Sequence Collector} skill in the game.
551 540  
552   -\begin{table}[h]
553   -\caption{Statistics on the average sequence length of players with and without the {\em Sequence Collector} skill}\label{tab_seqCollector}
554   -\begin{center}
555   -\begin{tabular}{cccccc}\hline
556   - & Nb Players & Min & Max & Median & Average\\
557   -With & 22 & 2.33 & 9 & 5.04 & 5.04\\
558   -Without & 40 & 2.39 & 9.05 & 5.40 & 5.31\\\hline
559   -\end{tabular}
560   -\end{center}
561   -\end{table}
  541 +\begin{figure}[htbp]
  542 + \begin{center}
  543 + \includegraphics[width=\halfWidth]{Figs/boxplot_SC_seqLength.pdf}
  544 + \vspace{0cm}
  545 + \caption{Boxplot of the average sequence length of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill.
  546 + }\label{fig_boxplotSC_seqLength}
  547 + \end{center}
  548 +\end{figure}
562 549  
563   -Table~\ref{tab_seqCollector} shows statistics on the average sequence length of the players with the {\em Sequence Collector} skill and the other players.
564   -We can observe here a pattern that is similar to the one of the {\em Color Expert} skill: players without the skill are, on average, selling sequences that
565   -are longer. Once again, this can be explained by the fact that selling long sequences is one of the two main goals of the game, and is one of the main components
  550 +We first compared the average sequence length of sequences built by players with the {\em Sequence Collector} skill and the ones built by the rest of
  551 +players~\ref{fig_boxplotSC_seqLength}. While the median value for the players without the skill (5.63) is a little bit larger than the one for the players
  552 +with the skill (5.12), the averages of both groups are actually similar (5.61 and 5.56 in the same order). Since the distribution of the average
  553 +sequence lengths were not normal (the Shapiro-Wilk test rejected the null hypothesis with $p = 0.0057$), we did a Mann Whitney's U test to compare the
  554 +medians of both groups. The test failed to reject the null hypothesis that the values were sampled from the same distribution ($p = 0.69$). Thus,
  555 +there is no evidence that the {\em Sequence Collector} skill helps players build longer sequences. This tends to confirm what we mentioned earlier (in Section
  556 +Testing hypothesis 1): the presence of the skills in general does not seem to affect the length of the sequences built by players.
  557 +Once again, this can be explained by the fact that selling long sequences is one of the two main goals of the game, and is one of the main components
566 558 of the scoring function.
567 559  
568   -Once again, we made the comparison between the players with at least two points in the {\em Sequence Collector} skill and all the others. Just like with the
569   -{\em Color Expert} skill, we observe that the players who really choose to specialize in this ability are doing better than the other players. As shown in
570   -Table~\ref{tab_seqCollector2}, the average sequence length is a little bit higher for players who put more than one skill point in {\em Sequence Collector}.
  560 +\begin{figure}[htbp]
  561 + \begin{center}
  562 + \includegraphics[width=\halfWidth]{Figs/boxplot_SC_nbCols.pdf}
  563 + \vspace{0cm}
  564 + \caption{Boxplot of the average number of colors in common of sequences built by players with (60 players) and without (83 players) the {\em Sequence Collector} skill.
  565 + }\label{fig_boxplotSC_nbCols}
  566 + \end{center}
  567 +\end{figure}
571 568  
572   -\begin{table}[h]
573   -\caption{Statistics on the average sequence length of players with at least two points in the {\em Sequence Collector} skill vs the others}\label{tab_seqCollector2}
574   -\begin{center}
575   -\begin{tabular}{cccccc}\hline
576   - & Nb Players & Min & Max & Median & Average\\
577   -$\geq$ 2 pts & 10 & 2.78 & 9 & 5.10 & 5.35\\
578   -Others & 52 & 2.33 & 9.05 & 5.11 & 5.19\\\hline
579   -\end{tabular}
580   -\end{center}
581   -\end{table}
  569 +We then compared the average number of colors in common of the sequences built by players with and without the {\em Sequence collector} skill~\ref{fig_boxplotSC_nbCols}.
  570 +The median value for the players without the skill (1.58) is 14\% lower than the one for the players with the skill (1.80). Since the distribution of the average
  571 +number of colors in common is not following a normal distribution (the Shapiro-Wilk test rejected the null hypothesis
  572 +with $p = 1.2E-7$), we used a Mann-Whitney's U test to compare the medians of the two groups and we found a significant effect of the presence
  573 +of this skill on the medians ($U = 3113$, $p = 0.01$, effect size $r = 0.21$). The {\em Sequence collector} skill is thus helping players
  574 +to build sequences with more colors, by allowing them to store unfinished sequences of multiple colors in the additional slots until they are able to complete them.
582 575  
583 576 \subsection{Testing hypothesis 3: the usefulness of the challenge system}
584 577  
... ... @@ -666,7 +659,7 @@
666 659 doing actions that are not specific to a certain subset of colors. Even if the market should be helpful in finding circles with the required
667 660 subset of colors, it seems highly probable that the players felt that this type of challenge was too hard and never tried to complete it.
668 661  
669   -\subsection{Testing hypothesis 4: relationship between total experience and percentage solved}\label{sect_hyp4}
  662 +\subsection{Testing hypothesis 4: relationship between total experience and percentage solved}
670 663 %Coming back on the 4 tests, total game xp vs percentage of problem solved
671 664 As mentioned in the Experiments section, the initial plan was to measure the impact of each feature by analyzing how much of the problem can be solved
672 665 by the players in each of the game sessions. Interestingly, we observed a larger than expected variance in the participants' skills which made it practically