Commit d0e750a66fb990e1c1772204bdf6c734bf72c99f

Authored by waldispuhl
1 parent 069c75880a
Exists in master

answer2refs.txt

Showing 1 changed file with 44 additions and 0 deletions Side-by-side Diff

answers2referees.txt View file @ d0e750a
  1 +First of all, thank you all for the constructive remarks. We want here to briefly go through all the comments and offer some clarifications if need be.
  2 +
  3 +- Reviewer 1:
  4 +
  5 +1) We totally agree that the introduction is not comprehensive enough. We ended up missing some time before the deadline to write the introduction we really wanted to write. Thanks a lot for the reference; it is indeed a good starting point.
  6 +
  7 +2) This is an important suggestion. We will definitely state all the questions and hypotheses in the introduction. This will help a lot to clarify the goals of this paper, which is a remark that was made by the other reviewers as well.
  8 +Just to clarify, the questions we are trying to answer in this paper (and hypotheses) are:
  9 + 1. Is a market system helping the players to build better solutions? (hypothesis: yes)
  10 + 2. Is a skill system useful to orient the players into doing specific actions that are beneficial to the game and other players? (hypothesis: yes)
  11 + 3. Is a challenge system effective in encouraging the players to do a specific action in the game? (hypothesis: yes)
  12 + 4. Are the collected solutions better when all the 3 features are on in a game session, independently of the players' skills? (hypothesis: yes)
  13 +
  14 +3) Clearly stating all the variables (dependent and independent) is a good suggestion.
  15 +Total XP and the percentage of solution found were measured as a way to answer question #4. We agree that it was unclear, and this could be a consequence of not stating the research questions clearly in the introduction.
  16 +
  17 +4) Thank you for giving suggestions on the statistical tests we can do. We are starting to work on them.
  18 +
  19 +5) Testing if the results are task independent is a great idea. Since major changes to the interface would probably be necessary to represent a different task, this is something that we could look into for a future project.
  20 +
  21 +6) We will remove sentences that feel like diary entries. The sentence on the cancellations was also very unclear. We were only trying to explain that not all 10 players were new players. In all of the five sessions, we had one or two experienced player(s) filling in for new players who couldn't be there on time.
  22 +
  23 +- Reviewer 2:
  24 +
  25 +Thanks for the interesting remark. Indeed, we did not write enough about comparing the performance of algorithms versus the performance of players for this graph problem.
  26 +Actually, for a very limited number of colors (like 6 in our tests), an exact algorithm can solve the problem in seconds. Increasing the number of colors makes the problem harder for computers, but for players also. There is also a limit to the number of colors that can be displayed clearly in a circle.
  27 +What we should have mentioned in the paper is that the instance of the problem we are trying to solve is too well-defined and structured to take advantage of all the human capabilities. If we had used, for example, approximate symbols or handwritten digits instead of colors, then the ability of human interpretation would have been extremely useful and humans could have performed better than computers.
  28 +In our case, we were satisfied with having a problem that was structured enough so that we could easily calculate the optimal solution and evaluate the performance of the players depending on what features were on or off and also the effect of the different features on the quality of solutions.
  29 +
  30 +- Reviewer 3:
  31 +
  32 +2nd paragraph:
  33 + - As was mentioned by the second reviewer, it is true that we did not write a lot about comparing the performance of the players with the performance of computers on this problem (see comments above for details on that).
  34 + - You mention a very important aspect of the game, which is competitiveness between the players. We agree that some biases could come from the competitive nature of the game. At the same time it is also a powerful motivation factor that we are trying to take advantage of. The different features of the game are implemented in such a way that the players' action are enhancing collaboration and that it is almost impossible to hurt another player.
  35 + - It is true that it can be hard for a player to find a complete solution on his own. The super circles remove the apparent limit of 10 circles maximum in a solution (by merging together a solution of length 10 and putting in back in circulation as one circle). The market is also a new approach that we proposed to help with that, but as you mentioned, a player is never forced to sell a circle that is very important to another player. The player who really needs a circle has to raise his bidding price in order to encourage the other players to sell. Players who choose the Master Trader skill that gives bonus money for selling are more likely to sell a lot because it can be a winning strategy for them. We chose to have a system based on choice rather than on forcing players to do actions, as it is less restraining for the players.
  36 + - About the random nodes comment, it made us realize that we forgot to mention clearly that there is a fixed number of distinct circles (nodes) in every game. The set of distinct circles is determined by the nodes of the graph that was generated for the study. However, the game creates copies of the circles when all the distinct circles are already in circulation in the game.
  37 +
  38 +3rd paragraph:
  39 + - Thanks a lot for pointing out that the objective of the paper is not mentioned clearly. Improving the introduction by adding related work and clearly stating the questions and hypotheses will clarify a lot of things. We would say that the general goal is to propose an approach (with different game mechanics) to efficient group puzzle solving. To our knowledge, this is the first time that a market system (to exchange data) coupled with skills and challenges (to guide the players) are proposed as an approach for collaborative solving in a human computing video game.
  40 +
  41 +4th paragraph:
  42 + - Thank you for the suggestions about the game interface. We will add inset labels as you proposed. Thanks for suggesting to add more explanations about the motivation behind the design choices. In this rebuttal, we already gave more insight into our guiding intentions; we can easily add that to the paper too.
  43 +
  44 +Finally, thank you for the list of typos.