Some tips for addressing peer reviews

A few thoughts that I shared with students

Mindset

Please do assume the best intentions of the reviewers and don't dismiss the reviews too quickly. Not properly understanding the main point, or not putting in enough effort to understand the reviews, is the most common pitfall!

It is all too common to blame Reviewer 2. Although there are indeed cases where the reviewer does not have any clue, in most cases, the reviewer might have actually read (!) and understood (!) the paper, and what they were trying to say may indeed be insightful. They might have to write it too quickly or be in a terrible mood when they were writing the review. So, if I can only give you one piece of advice, this will be it: try to really understand what the reviewers meant.

Do you think the reviewer totally missed a key point of the paper? Think again. Maybe they got the point, but they still said that. Why? Or, maybe the paper was not really written clearly enough to communicate the main point, and you'd need to seriously re-evaluate your paper. Often, by really trying to be in the reviewer's shoes, you can understand the reasons behind their comments and find better ways to address their comments rather than talking past each other.

I'd even argue that this — whether you understand where the reviewer is coming from and why they say certain things before sending the rebuttal or not — creates one of the most important edges in the peer review process. Do you keep failing in contentious review rounds? Then maybe, just maybe, you are dismissing the reviewers too quickly and not spending enough effort to understand them. Yes, there are times when you should try to disqualify the reviewer, but that's not very common.

Second, stay positive. It is common that the review feels impossible to address, initially. But, even the harshest reviewers can change their minds if you can provide a robust, convincing, and polite response. Even seemingly insurmountable comments can be addressed, more often than initially seemed. Remember that writing a rebuttal feels so much harder at the beginning but gets easier and easier as you carefully understand and address comments one by one. It is like a big puzzle, once you find solutions to some of the comments, that makes the whole thing much easier.

Third, stay polite and civil. Trying to disqualify a reviewer is the last resort and even that can be done in a civil way, without making the reviewer your enemy. If a reviewer is completely unqualified, make it evident to the editor by crafting a strong, yet polite response. Disagree with evidence; don't disparage. If the reviewer is rude, your polite rebuttal will highlight the rudeness even more evidently.

Finally, consider delivering more than what is requested or needed. Demonstrating something completely beyond possible doubts can save time (removing the necessity of further rounds of reviews) and can change reviewer's mind more effectively.

How to

First, you need to copy & paste every comment into your rebuttal document. Leaving out any comments — even a simple summary of the paper — is a no no. A reviewer said that the paper is well-written? Great! Copy that and express your gratitude for acknowledging the merit of your paper! Break each comment down by individual points and sketch how they can be addressed. Think about whether it can be done by revision/clarification or requires new experiments. You want to figure out what to do with the latter quickly because they require more discussion and time.

Share this document with all co-authors and, if necessary, have a meeting to discuss the strategies. Iterate. Run some experiments, draft a rebuttal, share & discuss, and repeat.

Try to delay the editing of the manuscript as much as possible. Often, the number of edits to the manuscript is less than what you might initially assume to be. By working only on the rebuttal document, you can avoid the issue of duplicate effort and divergence between the rebuttal and manuscript. As long as the revision is not extensive, just work on the paragraph in the rebuttal (before vs. after) and apply the edits to manuscript when the rebuttal is finalized. Having said that, if extensive revision is needed, it may be better to work on the manuscript to make it as coherent as possible and then bring the changes to the rebuttal. Be sure to keep a copy of the submitted version so that you can keep track of changes. Don't just rely on version control system like git; create separate folders for each version explicitly. Every major version — arXiv preprint version, each submission and resubmission, etc. — should be stored separately.

Often, multiple reviewers make similar comments. Don't be afraid of duplicating some responses. Most reviewers care about what they commented on the most, and will thus read the response to their own review most carefully. Referring to other sections makes it harder to follow the response. So, as long as you are not duplicating multiple pages of response, it is totally ok, or sometimes even preferable, to duplicate the response. Having said that, if the comment/rebuttal is pivotal and the rebuttal requires substantial text & figures (say more than a page or so), it is a good idea to extract that as a separate section, ideally before going into the point-by-point rebuttal for each reviewer.

When the rebuttal is taking shape, start drafting the cover letter, for the editor. The cover letter is an important part of the rebuttal. It should provide a succinct summary of major comments and how they are addressed. Thank the editor and reviewers for their effort, and then summarize the major points raised and how they are addressed in the rebuttal.

Feel free to use & improve my basic LaTeX template for rebuttal: https://github.com/yy/rebuttal-template