Commit 7822392291513f68ee05d94090f7d3018bc6577f

Authored by waldispuhl
1 parent d0e750a66f
Exists in master

answer2refs.txt

Showing 1 changed file with 43 additions and 16 deletions Side-by-side Diff

answers2referees.txt View file @ 7822392
1 1 First of all, thank you all for the constructive remarks. We want here to briefly go through all the comments and offer some clarifications if need be.
2 2  
  3 +
  4 +*** OLIVER: You could mention here that our goal is to quantify the computing power of markets and mechanism to control crowds. This should go to the introduction. ***
  5 +
3 6 - Reviewer 1:
4 7  
5   -1) We totally agree that the introduction is not comprehensive enough. We ended up missing some time before the deadline to write the introduction we really wanted to write. Thanks a lot for the reference; it is indeed a good starting point.
  8 +1) We totally agree that the current introduction is not comprehensive enough and misses references to several important prior contributions. We ended up missing some time before the deadline to write the introduction we really wanted to write. Thanks a lot for the reference; it is indeed a good starting point.
6 9  
7   -2) This is an important suggestion. We will definitely state all the questions and hypotheses in the introduction. This will help a lot to clarify the goals of this paper, which is a remark that was made by the other reviewers as well.
  10 +2) This is an important suggestion and we will definitely state clearly all the questions and hypotheses in the introduction. This will help a lot to clarify the goals of this paper, which is a remark that was made by the other reviewers as well.
8 11 Just to clarify, the questions we are trying to answer in this paper (and hypotheses) are:
9 12 1. Is a market system helping the players to build better solutions? (hypothesis: yes)
10 13 2. Is a skill system useful to orient the players into doing specific actions that are beneficial to the game and other players? (hypothesis: yes)
11 14 3. Is a challenge system effective in encouraging the players to do a specific action in the game? (hypothesis: yes)
12 15 4. Are the collected solutions better when all the 3 features are on in a game session, independently of the players' skills? (hypothesis: yes)
13 16  
14   -3) Clearly stating all the variables (dependent and independent) is a good suggestion.
15   -Total XP and the percentage of solution found were measured as a way to answer question #4. We agree that it was unclear, and this could be a consequence of not stating the research questions clearly in the introduction.
  17 +*** OLIVER: Can you say that your work confirm all these hypothesis? If yes, say it so :) ***
16 18  
17   -4) Thank you for giving suggestions on the statistical tests we can do. We are starting to work on them.
18 19  
19   -5) Testing if the results are task independent is a great idea. Since major changes to the interface would probably be necessary to represent a different task, this is something that we could look into for a future project.
  20 +3) We agree that our the description of the variables needs to be improved. Total XP and the percentage of solution found were measured as a way to answer question #4. This confusion could be a consequence of the lack of a clear description the research questions in the introduction. We guarantee the referee that all these points will be clarified in the final version.
20 21  
21   -6) We will remove sentences that feel like diary entries. The sentence on the cancellations was also very unclear. We were only trying to explain that not all 10 players were new players. In all of the five sessions, we had one or two experienced player(s) filling in for new players who couldn't be there on time.
  22 +*** OLIVER: It is good to indicate after each item which action you will take to address the referee’s remark ***
22 23  
  24 +4) Thank you for this important suggestion. We found that basic statistics reported in the paper were sufficient at this stage to support the hypothesis of this work (see above). Nonetheless, we agree with the referee that deeper statistical analysis will help to strengthen the conclusion and impact of this paper. This material will be added to a revised version.
  25 +
  26 +*** OLIVIER: Since we add nothing, can we say that our current data are ok for now but that more will be added later? Revise my text if you don’t like it ***
  27 +
  28 +5) We thank the referee for this useful suggestion. Since major changes to the interface would probably be necessary to represent a different task, this is something that we are willing to investigate into future projects.
  29 +
  30 +6) We will remove sentences that feel like diary entries. We initially added these details to be completely transparent on the process.
  31 +
  32 +
  33 +[The sentence on the cancellations was also very unclear. We were only trying to explain that not all 10 players were new players. In all of the five sessions, we had one or two experienced player(s) filling in for new players who couldn't be there on time.]
  34 +
  35 +*** OLIVIER: I don’t think the part between brackets is useful. You can just remove these sentences. and say you just wanted to be transparent ***
  36 +
23 37 - Reviewer 2:
24 38  
25   -Thanks for the interesting remark. Indeed, we did not write enough about comparing the performance of algorithms versus the performance of players for this graph problem.
26   -Actually, for a very limited number of colors (like 6 in our tests), an exact algorithm can solve the problem in seconds. Increasing the number of colors makes the problem harder for computers, but for players also. There is also a limit to the number of colors that can be displayed clearly in a circle.
27   -What we should have mentioned in the paper is that the instance of the problem we are trying to solve is too well-defined and structured to take advantage of all the human capabilities. If we had used, for example, approximate symbols or handwritten digits instead of colors, then the ability of human interpretation would have been extremely useful and humans could have performed better than computers.
28   -In our case, we were satisfied with having a problem that was structured enough so that we could easily calculate the optimal solution and evaluate the performance of the players depending on what features were on or off and also the effect of the different features on the quality of solutions.
  39 +Thanks for the interesting insight. Indeed, we did not write much about comparing the performance of algorithms versus the performance of players for this graph problem. The reason is that for a very limited number of colors (here 6), an exact algorithm can solve the problem quickly.
29 40  
  41 +In fact, we intentionally use this situation. Our capacity to calculate a complete solution was a prerequisite for us since our goal is to fully characterize the computational efficiency of a market system (e.g. percentage of solution found by humans).
  42 +
  43 +We should have mentioned in the paper is that the tasks we are trying to solve are (intentionally) well-defined and do not take advantage of all human capabilities. Nonetheless, to put this work in perspective, once the computing performance of the markets are evaluated, we can replace the color matching tasks with more complex task (e.g. handwritten digits instead of colors) in order to make a better use of humans skills.
  44 +
  45 +As noted by other reviewers too, the confusion may stem from an insufficient description of the research questions in the introduction (See answer to Ref. #1). We will ensure to clarify all these aspect in a revised introduction.
  46 +
  47 +
  48 +*** OLIVIER: We should do our best to convince referee 2 & 3 on this point. ***
  49 +
  50 +
30 51 - Reviewer 3:
31 52  
32 53 2nd paragraph:
33   - - As was mentioned by the second reviewer, it is true that we did not write a lot about comparing the performance of the players with the performance of computers on this problem (see comments above for details on that).
  54 + - As was also mentioned by the ref #2, it is true that we did not write enough about comparing the performance of the players with the performance of computers on this problem. The main reason for that is that we are here more interested in evaluating the absolute performance of markets to solve this problem rather than ‘beating the computer’. In fact, we intentionally used a NP-complete problem for which we can easily calculate the solution in order to estimate the absolute performance of the market (see also comments above).
  55 +
34 56 - You mention a very important aspect of the game, which is competitiveness between the players. We agree that some biases could come from the competitive nature of the game. At the same time it is also a powerful motivation factor that we are trying to take advantage of. The different features of the game are implemented in such a way that the players' action are enhancing collaboration and that it is almost impossible to hurt another player.
  57 +
  58 +*** OLIVIER: Again, you can say that you will clarify this in the text. ***
  59 +
35 60 - It is true that it can be hard for a player to find a complete solution on his own. The super circles remove the apparent limit of 10 circles maximum in a solution (by merging together a solution of length 10 and putting in back in circulation as one circle). The market is also a new approach that we proposed to help with that, but as you mentioned, a player is never forced to sell a circle that is very important to another player. The player who really needs a circle has to raise his bidding price in order to encourage the other players to sell. Players who choose the Master Trader skill that gives bonus money for selling are more likely to sell a lot because it can be a winning strategy for them. We chose to have a system based on choice rather than on forcing players to do actions, as it is less restraining for the players.
36   - - About the random nodes comment, it made us realize that we forgot to mention clearly that there is a fixed number of distinct circles (nodes) in every game. The set of distinct circles is determined by the nodes of the graph that was generated for the study. However, the game creates copies of the circles when all the distinct circles are already in circulation in the game.
37 61  
  62 +*** OLIVIER: Can you say that you have multiple copies of the SNP in the game to avoid that one player blocks everything? ***
  63 +
  64 +
  65 + - About the random nodes comment, we omitted to clearly mention that there is a fixed number of distinct circles (nodes) in every game. The set of distinct circles is determined by the nodes of the graph that was generated for the study. However, the game creates copies of the circles when all the distinct circles are already in circulation in the game.
  66 +
38 67 3rd paragraph:
39 68 - Thanks a lot for pointing out that the objective of the paper is not mentioned clearly. Improving the introduction by adding related work and clearly stating the questions and hypotheses will clarify a lot of things. We would say that the general goal is to propose an approach (with different game mechanics) to efficient group puzzle solving. To our knowledge, this is the first time that a market system (to exchange data) coupled with skills and challenges (to guide the players) are proposed as an approach for collaborative solving in a human computing video game.
40 69  
41 70 4th paragraph:
42 71 - Thank you for the suggestions about the game interface. We will add inset labels as you proposed. Thanks for suggesting to add more explanations about the motivation behind the design choices. In this rebuttal, we already gave more insight into our guiding intentions; we can easily add that to the paper too.
43   -
44   -Finally, thank you for the list of typos.