In the last month I wrote/edited 24 author responses to EC and AAAI reviews, and read 18 responses as an EC PC member and 45 responses as a AAAI SPC member, for a grand total of 87 author responses (counting three responses per paper). So you can imagine that author responses have been on my mind.
Author responses are a proud tradition in AI conferences. The theory side of the EC community seems to dislike them, and the weird result is that EC has author responses only on even years (when there is an AI PC co-chair). Debating the advantages and (nonexistent) disadvantages of author responses is a fun and popular pastime, but right now I want to do something a bit different: discuss their strategic aspects.
If you have two negative reviews then you can start studying the call for papers of your favorite second-tier conference, so let’s focus on the case where you have two positive reviews and a single negative review. The correct response of course depends on how negative the latter review is.
Case 1: the negative review is just below borderline (overall score of 6 in EC or 5 in AAAI/IJCAI). The goal is to gently nudge the negative reviewer upwards with high probability. Here I am pleased to announce that, after years of trial and error, I have identified the perfect strategy. The first step is to convince yourself (and your coauthors) that the reviewer is supremely wise. The second step is to respond by agreeing (honestly, see step 1) with everything the reviewer says and committing to a revision that addresses the reviewer’s main concern. A subtle obstacle is that conference papers should be accepted roughly as-is, hence this strategy requires a delicate balancing act between promising a revision and making the case that the revision is minor. The trick is to clearly point out exactly which paragraphs you plan to revise.
Case 2: the negative review is way below borderline. Ah, the twilight zone of author responses; here there are no perfect strategies, it’s just about playing the odds. Since we are assuming that your paper is awesome, this case implies a reviewer who completely misunderstands the paper, or an outlandish review. In AAAI/IJCAI, for example, you occasionally get the random PC member who worked on automated theorem proving in the Eighties and checked the game theory keyword because he really loves Super Mario Bros.
The first strategy is to politely point out to the reviewer why he/she totally misunderstands your paper. From my experience, this has a reasonably good chance of bumping the reviewer’s score upwards by two points. Strangely enough, two also seems to be an upper bound on the possible benefit; even after realizing the error of their ways reviewers never seem to switch from negative to positive, they get stuck at borderline. Nevertheless, this common strategy seems to work well if the reviewer’s initial overall score is 4.
The second (more desperate) strategy is to share your honest thoughts about the dubious quality of the negative review; the goal of this gambit is to get an additional review, ideally as a replacement. One possibility is to do this via a side channel to the PC chairs, but that seems a bit sneaky. I have only attempted the direct approach (via the author response) once. The outcome is best described using a useful Hebrew Idiom: “the operation succeeded but the patient died” (an additional review was given but the paper was rejected). Next time I think I’ll try the sneaky approach.
Finally, one EC SPC did not appreciate my admittedly blunt attempt to point out a perceived discrepancy between reviewers’ verbal recommendations and overall scores. In a AAAI response, my coauthor John Lai took a more diplomatic approach for the same issue: “We noticed a possible discrepancy between the verbal recommendation (…) and overall score (…) and wanted to check that this was intended.” It occurred to me later that a slight variation of this statement could provide a foolproof author response to any negative review: “We noticed a possible discrepancy between the review and the quality of our paper.” Do let me know if it works.
Turing's Invisible Hand 