Commit 063262b6b0b1a3d9c2ee4f15ddc612ea0d858da2

Authored by Olivier
1 parent 7822392291
Exists in master

Final version sent

Showing 1 changed file with 43 additions and 0 deletions Side-by-side Diff

answers2referees_final_final.txt View file @ 063262b
  1 +Thank you all for the constructive remarks. We will briefly go through all the comments and offer some clarifications.
  2 +
  3 +- Reviewer 1:
  4 +
  5 +1) We totally agree that the current introduction is not comprehensive enough and misses references to several important prior contributions. We ended up lacking time to write the introduction we wanted to write. Thanks for the reference; it is a good starting point.
  6 +
  7 +2) This is an important suggestion. We will definitely state clearly all the questions and hypotheses in the introduction. This will help a lot to clarify the goals of this paper, which is a remark that was made by the other reviewers as well.
  8 +The questions we are trying to answer in this paper (and hypotheses) are:
  9 +1. Is a market system helping the players to build better solutions? (hypothesis: yes; confirmed)
  10 +2. Is a skill system useful to orient the players into doing specific actions? (hypothesis: yes; confirmed)
  11 +3. Is a challenge system effective in encouraging the players to do a specific action? (hypothesis: yes; confirmed)
  12 +4. Are the collected solutions better when all the 3 features are on in a game session, independently of the players' skills? (hypothesis: yes; refuted)
  13 +
  14 +3) We agree that the description of the variables needs to be improved. Total XP and % of solution found were measured as a way to answer question #4. This confusion could be a consequence of the lack of a clear description of the research questions. We guarantee that all these points will be clarified in the final version.
  15 +
  16 +4) Thank you for this important suggestion. We found that basic statistics reported in the paper were sufficient at this stage to answer the questions (see above). Nonetheless, we agree that deeper statistical analyses will help to strengthen the conclusion and impact of this paper. This material will be added to a revised version.
  17 +
  18 +5) We thank the referee for the suggestion. Since major changes to the interface would probably be necessary to represent a different task, this is something that we are willing to investigate in future projects.
  19 +
  20 +6) We will remove sentences that feel like diary entries. We initially added these details to be completely transparent on the process.
  21 +
  22 +- Reviewer 2:
  23 +
  24 +Thanks for the interesting insight. Indeed, we did not write much about comparing the performances of algorithms vs players for this graph problem. The reason is that for a very limited number of colors (here 6), an exact algorithm can solve the problem quickly.
  25 +In fact, we intentionally use this situation. Our capacity to calculate a complete solution was a prerequisite since our goal is to characterize the computational efficiency of a market system.
  26 +
  27 +We should have mentioned in the paper that the tasks we are trying to solve are (intentionally) well-defined and do not take advantage of all human abilities. Nonetheless, once the computing performance of the markets are evaluated, we can replace the color matching tasks with more complex tasks (e.g. handwritten digits instead of colors) in order to make a better use of human skills.
  28 +
  29 +As noted by other reviewers too, the confusion may stem from an insufficient description of the research questions (see answer to Rev. 1). We will ensure to clarify all these aspects in a revised introduction.
  30 +
  31 +- Reviewer 3:
  32 +
  33 +As was also mentioned by Rev. 2, it is true that we did not write enough about comparing the performances of computers vs players on this problem. The main reason is that we are more interested in evaluating the absolute performance of markets to solve this problem rather than ‘beating the computer’ (see also comments above).
  34 +
  35 +You mention a very important aspect of the game, which is competition between players. We agree that some biases could come from the competitive nature of the game. At the same time it is a powerful motivation factor that we want to take advantage of. The different features of the game are implemented in such a way that the players' actions are enhancing collaboration and that it is almost impossible to hurt another player. This will be clarified in the revised version.
  36 +
  37 +It is true that it can be hard for a player to find a complete solution on his own. The super circles remove the apparent limit of 10 circles maximum in a solution (by merging a solution of length 10 and putting it back in circulation as one circle). The market is also a new approach that we proposed to help with that, but as you mentioned, a player is never forced to sell a circle. The player who really needs a circle has to raise his bidding price in order to encourage the other players to sell. Players who choose the skill that gives bonus money for selling are more likely to sell a lot as it can be a winning strategy for them. We chose to have a system based on choice that is less restraining for the players.
  38 +
  39 +About the random nodes comment, we omitted to clearly mention that there is a fixed number of distinct circles (nodes) in every game. The set of distinct circles is determined by the nodes of the graph that was generated. However, the game creates copies of the circles when all the distinct circles are already in circulation in the game.
  40 +
  41 +Thanks for pointing out that the objective of the paper is not stated clearly. As mentioned earlier, improving the introduction will make things clearer. In short, the general goal is to propose an approach (with different game mechanics) to efficient group puzzle solving. To our knowledge, this is the first time that a market system (to exchange data) coupled with skills and challenges (to guide the players) are proposed as an approach for collaborative solving in a human computing game.
  42 +
  43 +Thank you for the suggestions about the game interface. We will add inset labels as you proposed. We will also add more explanations about the motivation behind the design choices. In this rebuttal, we already gave more insight into our guiding intentions; we will add that to the paper too.