Peer Review

You have been asked to peer review for a journal (or grant applications, or conference submissions, or...) in your field.  Great!  You’re providing a service to your field, and this experience can be added to your CV. 

But.  

You’ve only ever done peer review in class on classmates’ papers. 

How is peer review for a journal different?  How do you do it?

This page contains several How-To resources from well-known journal publishers such as Nature, Taylor & Francis, PLOS, and Wiley, as well as a few resources created by researchers for researchers.  There are several online written guides as well as workshops or classes you could sign up for. Additionally, we try to highlight similarities and differences between peer reviewing journal article sand other types of scholarly materials that undergo peer review. Finally, we offer some resources to help you, as an author, respond to peer review in professional and constructive ways.

This page is heavy on the resources for reviewers in science.  But the best practices don't vary much by discipline. 

The editor or review chair who invited you to review should share any guidelines, rubrics, scoring sheets, and expectations for what to look for and how to communicate.  Many scholarly publishers have pages for instructions to reviewers, much like instructions for authors when you submit a manuscript (see below for some examples).  Many journals will have more specific instructions.  If you are reviewing competitive applications, such as for grants or conference presentations, read the instructions for submission as well as any scoring tools provided to reviewers.  If the person administering the review process does not volunteer them, don't hesitate to request them.       

A 2009 editorial in American Naturalist proposed the Golden Rule of Reviewing: review for others as you would have others review for you. Say yes to reviewing whenever your duties and schedule allow; provide a thorough, fair, and constructive critique of the work; and do it at your first opportunity regardless of the deadline (McPeek et al.).

from "Behind the Scenes in Academic Publishing: A Closer Look at Peer Review" Guest Commentary by Yara Abdou in July 2020 for the ASCO Connection blog (American Society for Clinical Oncology)

Some prompts to answer during your second or third pass (adapted primarily from the Sense About Science guide "Peer Review: The Nuts and Bolts"):

• Is the research question or thesis clear?
• Was the approach appropriate?
• Is the study design, methods, and analysis appropriate to the question or thesis?
• Is the work innovative or original?
  • Does it challenge existing paradigms or add to existing knowledge?
  • Does it develop novel concepts?
  • Does it matter?
• Are the methods described clearly enough for other researchers to replicate? / Are the cited sources of reputable provenance?
• Are the methods of statistical analysis and level of significance appropriate? / Do the arguments appropriately develop the thesis?
• Could presentation of the results be improved and do they answer the question? / Do the researchers present a compelling narrative and conclusion?

Some tips for writing the final review, synthesized from sources on this page:

• Be constructive, professional, and specific. 
  • Even if your determination is that the paper is not of sufficient quality to accept, your goal in providing peer review is to improve the work.  If your comments are uniformly or excessively negative or unhelpful, the author(s) are unlikely to make use of it.  Comment on things that are done well and offer suggestions for improvement.
    • Don't mistake saying something negative for being rude.  You were chosen to review due to your expertise.  Share your knowledge and challenge the author(s) to improve their work by being explicit with constructive criticism where warranted.
  • This is a professional interaction and your comments should reflect that.  Regardless of the style of peer review (double blind, open, etc.) your identity may become known to the author down the line and any lack of professionalism on your part will be taken into account in future interactions.  Plus, what goes around, comes around: strive to uplift the overall civility and helpfulness in peer reviewing with your contribution.  
  • "I don't understand what you mean" can be taken a number of ways.  "This sentence is overly complex and makes it hard to follow your argument.  Can you break it down?" is helpful.

The Committee on Publication Ethics (COPE) has several resources for familiarizing both reviewers and editors of journals with ethical standards and best practices in the peer review process.  This page below provides key takeaways for reviewers, as well as a link to their full report,  "Ethical guidelines for peer reviewers, COPE guidance." 

We hope you will also explore their other resources on peer review.

The Council of Science Editors, an international membership organization for editorial professionals publishing in the sciences, maintains a live document entitled "Recommendations for Promoting Integrity in Scientific Journal Publications."  Section 2.3 focuses on Reviewer Roles and Responsibilities, including ethical considerations.

The Association of Teachers of Technical Writing (ATTW) is "an active organization of teachers, researchers, and teacher-practitioners of technical communication...[who] work in graduate and undergraduate programs in technical communication, media, engineering, rhetoric, writing studies, and English, among other complementary research programs."  The creation of a heuristic for anti-racist scholarly reviewing practices emerged from challenges made by Angela Haas in her 2020 ATTW address “Call to Action to Redress Anti-Blackness and White Supremacy” and Miriam Williams and Natasha Jones, ATTW vice-president and fellow (respectively), in their 2020 ATTW blog post, “A Just Use of Imagination.” In these calls, these scholars asked members of the field to engage seriously with anti-racist change-making. The document below responds to their calls by focusing on academic reviewing processes.

These pages from publishers offer advice for providing a helpful, professional, and timely review for their journals.  Individual journals may have additional, more specific advice.  The person extending the invitation to review should include a link or attachment with this information for their publication.  However, sometimes that is only provided after you agree to review, and this information can be helpful in making your decision.  If you have received an invitation to review from a different publisher, see if you can find their page of advice.  Feel free to reach out to your librarian for help.

The Best Practices box above touches on the consistent aspects of performing peer review regardless of the material you are reviewing, be it a journal article, book or book chapter, conference proposal, or grant application.  Here we highlight some aspects in which reviewing varies by type:

Book/Book Chapter
Reviews of scholarly books or book chapters are much less likely to be double-blind.  Often, part of the reviewer's task is to evaluate the author's credentials and ability to speak to the topic they have chosen.  Sometimes your task as a reviewer is to assess a book proposal, wherein this aspect of evaluation takes on a larger role in your review.

Reviewing for books or book chapters is usually a more iterative process than journal article reviewing, and correspondingly takes longer.

Grant Application
In contrast to books, grant application peer review is often conducted on a very strict timeline.  Review may be needed within weeks of the date the materials are provided to reviewers.  Peer reviewers will evaluate several applications within that time frame.

Reviews of grant applications center the goals of the grant and granting agency to a higher degree than most journals do for article submissions.  How does this grant advance the mission or aims of the granting agency?  How well has the applicant aligned their work with the aims of the opportunity?

Since the proposal includes actions and deliverables, reviewers must also assess feasibility, both for the appropriateness of resources requested to the task and for the applicant's ability to complete the proposal.  Grant reviews are almost guided by a scoring rubric, so that applications are considered on consistent merits, rather than comparing applications.

Peer reviewers often meet together as a panel with administrators from the granting agency to discuss and re-score the highest-scoring applications.  Discussing applications with other reviewers can highlight strengths or weaknesses that were not clear in a solo assessment.  Not all grant application peer review processes include a panel discussion, but many do.

Here's a short video of the process for peer reviewers of grant applications submitted to the Institute of Education Sciences:

Despite the use of scoring rubrics and rigorous conflict of interest disclosures, peer reviewers must make many judgement calls in the course of rating an application, and apply their own experience and expertise to their decisions.  This can lead to a wide range of scores and raise the question of bias among reviewers.  These two scholarly articles come to somewhat different conclusions on the topic:

Conference Proposals
One big difference between reviewing conference proposals and journal articles is that it is extremely rare that a conference proposal author will be asked to revise and resubmit.  At best, they may be encouraged to change their submission to a less rigorous type of conference material, such as a poster.  Most often, they are rejected for the current year but encouraged to incorporate reviewers' feedback and resubmit the following year.  Most conferences just receive too many applications for a back-and-forth workflow to be feasible.

Much like grants, reviewers for conference proposals are working on a tight deadline, and reviewing several proposals in that time frame.  

And More
The traditional journal article has proliferated into a plethora of content-specific article or material types, such as data articles and datasets, methods articles, etc.  One example:

In addition to types of peer review, as described on the home page of this guide, peer review can take place at different stages in the peer review process, such as the registered report process (primarily in medicine).  Quick overview of that process:

If you are asked to review for a different type of scholarly work, please consult with your advisor, mentors in your field, and your librarian for advice and resources to help you decide whether a request is a good fit for you and to be the best reviewer you can be.

As a researcher, you will find yourself needing to respond to comments from peer reviewers of your own manuscript many times throughout your career.  Just as with performing peer review, how you conduct yourself in communication with the editor and the reviewers is a reflection on your professionalism and commitment to making the material the best it can be.  However, that doesn't mean that reviewers' advice should be followed without exception.  You are the expert on your own work.  The resources below offer advice on all aspects of your reply, from communicating changes made to your manuscript in response to reviewers' comments to how to push back on points that you think are stronger unedited.  As well as how to keep your cool while doing so!