Commit 5a5e6dd747a39acf86d7d7ecce80074dbda92f71

Authored by Olivier
1 parent 8a37ef08eb
Exists in master

Corrected some mistakes. Added one sentence: To our knowledge, this is the first…

… time a market system is proposed as a mechanism for collaborative solving in a human computing game.

Showing 1 changed file with 12 additions and 12 deletions Side-by-side Diff

rebuttal.txt View file @ 5a5e6dd
1 1 First of all, thank you all for the constructive remarks. We want here to briefly go through all the comments and offer some clarifications if need be.
2 2  
3   -- Assigned_Reviewer_1:
  3 +- Reviewer 1:
4 4  
5 5 1) We totally agree that the introduction is not comprehensive enough. We ended up missing some time before the deadline to write the introduction we really wanted to write. Thanks a lot for the reference; it is indeed a good starting point.
6 6  
7 7 2) This is an important suggestion. We will definitely state all the questions and hypotheses in the introduction. This will help a lot to clarify the goals of this paper, which is a remark that was made by the other reviewers as well.
8 8 Just to clarify, the questions we are trying to answer in this paper (and hypotheses) are:
9   - 1. Is a market system helping the players to build better solutions? (hypothesis: yes, it is)
10   - 2. Is a skill system useful to orient the players into doing specific actions that are beneficial to the game and other players? (hypothesis: yes, it is)
11   - 3. Is a challenge system effective in encouraging the players to do a specific action in the game? (hypothesis: yes, it is)
12   - 4. Are the collected solutions better when all the 3 features are on in a game session, independently of the players' skills? (hypothesis: yes, they are)
  9 + 1. Is a market system helping the players to build better solutions? (hypothesis: yes)
  10 + 2. Is a skill system useful to orient the players into doing specific actions that are beneficial to the game and other players? (hypothesis: yes)
  11 + 3. Is a challenge system effective in encouraging the players to do a specific action in the game? (hypothesis: yes)
  12 + 4. Are the collected solutions better when all the 3 features are on in a game session, independently of the players' skills? (hypothesis: yes)
13 13  
14 14 3) Clearly stating all the variables (dependent and independent) is a good suggestion.
15 15 Total XP and the percentage of solution found were measured as a way to answer question #4. We agree that it was unclear, and this could be a consequence of not stating the research questions clearly in the introduction.
16 16  
17 17  
18 18  
19 19  
20 20  
... ... @@ -20,26 +20,26 @@
20 20  
21 21 6) We will remove sentences that feel like diary entries. The sentence on the cancellations was also very unclear. We were only trying to explain that not all 10 players were new players. In all of the five sessions, we had one or two experienced player(s) filling in for new players who couldn't be there on time.
22 22  
23   -- Assigned_Reviewer_2:
  23 +- Reviewer 2:
24 24  
25 25 Thanks for the interesting remark. Indeed, we did not write enough about comparing the performance of algorithms versus the performance of players for this graph problem.
26 26 Actually, for a very limited number of colors (like 6 in our tests), an exact algorithm can solve the problem in seconds. Increasing the number of colors makes the problem harder for computers, but for players also. There is also a limit to the number of colors that can be displayed clearly in a circle.
27   -What we should have been mentioned in the paper is that the instance of the problem we are trying to solve is too well-defined and structured to take advantage of all the human capabilities. If we had used, for example, approximate symbols or handwritten digits instead of colors, then the ability of human interpretation would have been extremely useful and humans could have performed better than computers.
  27 +What we should have mentioned in the paper is that the instance of the problem we are trying to solve is too well-defined and structured to take advantage of all the human capabilities. If we had used, for example, approximate symbols or handwritten digits instead of colors, then the ability of human interpretation would have been extremely useful and humans could have performed better than computers.
28 28 In our case, we were satisfied with having a problem that was structured enough so that we could easily calculate the optimal solution and evaluate the performance of the players depending on what features were on or off and also the effect of the different features on the quality of solutions.
29 29  
30   -- Assigned_Reviewer_3:
  30 +- Reviewer 3:
31 31  
32 32 2nd paragraph:
33 33 - As was mentioned by the second reviewer, it is true that we did not write a lot about comparing the performance of the players with the performance of computers on this problem (see comments above for details on that).
34 34 - You mention a very important aspect of the game, which is competitiveness between the players. We agree that some biases could come from the competitive nature of the game. At the same time it is also a powerful motivation factor that we are trying to take advantage of. The different features of the game are implemented in such a way that the players' action are enhancing collaboration and that it is almost impossible to hurt another player.
35 35 - It is true that it is hard for a player to find a complete solution on his own. The market is a new approach that we proposed to help with that, but as you mentioned, a player is never forced to sell a circle that is very important to another player. The player who really needs a circle has to raise his bidding price in order to encourage the other players to sell. Players who choose the Master Trader skill that gives bonus money for selling are more likely to sell a lot because it can be a winning strategy for them. We chose to have a system based on choice rather than on forcing players to do actions, as it is less restraining for the players.
36   - - About the random nodes comment, it made us realize that we forgot to mention clearly that there is a fixed number of distinct circles (nodes) in every game. The set of distinct circles is defined by the nodes of the graph that was generated for the study. However, the game creates copies of the circles when all the circles are already in circulation in the game.
37   -
  36 + - About the random nodes comment, it made us realize that we forgot to mention clearly that there is a fixed number of distinct circles (nodes) in every game. The set of distinct circles is determined by the nodes of the graph that was generated for the study. However, the game creates copies of the circles when all the distinct circles are already in circulation in the game.
  37 +
38 38 3rd paragraph:
39   - - Thanks a lot for pointing out that the objective of the paper is not mentioned clearly. Improving the introduction by adding related work and clearly stating the questions and hypotheses will clarify a lot of things. We would say that the general goal is to propose an approach (with different game mechanics) to efficient group puzzle solving.
  39 + - Thanks a lot for pointing out that the objective of the paper is not mentioned clearly. Improving the introduction by adding related work and clearly stating the questions and hypotheses will clarify a lot of things. We would say that the general goal is to propose an approach (with different game mechanics) to efficient group puzzle solving. To our knowledge, this is the first time that a market system is proposed as a mechanism for collaborative solving in a human computing game.
40 40  
41 41 4th paragraph:
42   - - Thank you for the suggestions about the game interface. We will add inset labels as you proposed. Thanks a lot for suggesting to add more explanations about the motivation behind the design choices. In this rebuttal, we already gave more insight into our guiding intentions; we can easily add that to the paper too.
  42 + - Thank you for the suggestions about the game interface. We will add inset labels as you proposed. Thanks for suggesting to add more explanations about the motivation behind the design choices. In this rebuttal, we already gave more insight into our guiding intentions; we can easily add that to the paper too.
43 43  
44 44 Finally, thank you for the list of typos.