Including and valuing postdocs in peer review
This week, September 20-24 2021, brings together two of my favorite topics: peer review and postdocs. Peer Review Week 2021 “Explores the Role of Identity in Peer Review” and coincides with National Postdoc Appreciation Week. So in this blogpost let’s discuss the role that early career researchers (ECRs, such as postdocs) currently play in peer review, the challenges ECRs face, and ways forward to better value their participation in peer review. Joining me in this conversation is my colleague Dr. Rebeccah (Becki) Lijek at Mount Holyoke College, and I hope you’ll join us too through a Tweet Chat at noon U.S. EST on Tuesday, September 21st, 2021.
GARY: Becki, you and I have spent the last 3 years studying the participation of ECRs in peer review, largely looking at the phenomenon of co-reviewing. Co-reviewing is when ECRs perform peer review at the request of an invited reviewer, who is the person who was asked to review by the journal editor and is usually the ECRs’s research advisor. We found that co-reviewing is common practice because it trains the ECR and offloads labor from the busy faculty member. These data from our 2019 paper “Co-reviewing and ghostwriting by early-career researchers in the peer review of manuscripts” showed us some pretty concerning results. In practice, the participation of ECRs in peer review is mostly hidden from journals and journal editors, an issue not only for ECRs trying to be asked to carry out reviews in their own right, but a major issue for conflict-of-interest concerns, meaning we are effectively unable to establish how ethical the reviews are, given the lack of knowledge of who is actually doing them.
BECKI: That’s right. Additionally, a large proportion of ECRs are writing reviews and never receiving feedback on what they wrote - meaning that this isn’t even a proper training exercise.
We've realized how sparse training opportunities actually are, and how rarely evidence-based courses in peer review are made available (for more detail on this, see the Appendix of our comprehensive preprint). In short, ECRs are straining to find quality training in peer review of journal manuscripts.
GARY: This clashes spectacularly, in my mind, with the discussions of reviewer burnout, and the shortage of reviewers. How is it that I encounter so many ECRs struggling to find opportunities to carry out peer review, and simultaneously so many publishers and journal editors complaining of a lack of available reviewers?
I have personally encountered some discouraging comments in the scholarly publishing community that “postdocs shouldn’t be doing peer review” (like this opinion piece in The Scientist - and our rebuttal), and I want to take a moment to address the gatekeeping of intellectual labor which is taking place here. “Postdocs aren’t competent to peer review journal articles” isn’t a sensible claim, especially juxtaposed with claims of the skills provided by earning a PhD - are we really giving doctoral degrees to people incapable of critically evaluating another scientist’s work? Rather, I take the view that this is a defensive cultural attitude to protect control over intellectual labor - thinking and writing. Most practical labor in labs is entirely delegated to ECRs, so this comes across to me as an overly careful watch to protect intellectual labor such as manuscript authorship and peer review in both journals and funding agency study sections.*
BECKI: What defines a capable peer reviewer? At the recent FeedbackASAP workshop we hosted, participants discussed this very question and came to the decision that a reviewer is “anyone participating in science willing to think deliberatively, critically, and constructively about the work.” Note that career status is not a factor in this definition.
In fact, I don’t think this issue is restricted to ECRs. In talking with other faculty at liberal arts colleges, or research institutions that are considered less elite, there appear to be lower rates of invitations to peer review. The reasons for this are unclear: is it that editors tend to rely on personal networks, or name recognition within their field? Is it that names are drawn from corresponding authors on papers, which may appear more frequently on larger labs at research-intensive institutions? I hope in the discussions of identity in peer review this week, we hear more about how much issues with reviewer burnout are to do with the small subset of who is being asked, rather than the actual availability of capable reviewers. Crucially, it is important to know what data journals have on the diversity of their reviewer pools, including factors such as career stage, institution and country of location.
GARY: The effect of one's network seems to be a strong predictor of who gets asked to carry out a review. eLife carried out some experiments about including ECRs in their reviewer pool, and found that editors still tend to ask faculty they know, or whose names they recognize, to carry out the reviews. So including ECRs in review pools - but not asking them to review - is not fixing the problem. This has subsequently led them to experiment with trying to game databases to increase the likelihood of editors asking ECRs to carry out reviews.
Like me, you may have trauma from high school of being picked last for sports teams - it feels like this may also be happening here! It’s not that there aren’t enough capable reviewers, it’s that there aren't enough popular reviewers - and that is the definition of gatekeeping. And the gatekeeping is in name only and so doesn’t actually make sense, because since so many PIs delegate the labor to unnamed ECRs, the product is still co-authored by an ECR. So the gatekeeping is about the NAME on the review, not about the actual doing of the review.
BECKI: All this points to the considerable work that is required to increase the diversity of who is asked to carry out peer review, and do so in an inclusive and equitable manner that doesn’t rely on personal networks. Given that part of the issue arises from the general problem with how unclear journal publishing policies are, we have published recommendations to ECRs, reviewers, editors and journals on how the peer review system can better value the participation of ECRs. For example, by adjusting journal policies to explicitly endorse ethical co-reviewing practices or the inclusion of ECRs such as postdocs as invited reviewers in their own right.
GARY: I also want to point out the fantastic work in providing thoughtful training and genuine peer review experiences being carried out by the team at PREReview, who recently completed a pilot of a mentored peer-review training program specifically focusing on inclusive environments for ECRs. Such deliberate and inclusive work is going to be necessary to shift to a system that more naturally includes postdocs in peer review. And after all, including more postdocs in peer review will diversify the intersecting identities of peer reviewers (e.g. by race, gender, sexual orientation etc.), since the demographics of ECRs are more diverse than that of faculty (Gibbs et al., 2016).
BECKI: We plan to carry on working in this space by developing and testing a pedagogically-rigorous peer review curriculum, not only for its efficacy but also for its ability to increase a sense of belonging in peer review, and therefore, science. Postdocs struggle with realizing their scientific identity, I believe in part because of gatekeeping practices such as those keeping them from being directly invited by journals to participate in peer review. Teaching courses in peer review is one mechanism to make science more inclusive; level the playing field, so that quality training in peer review is available to all, not just those with mentors who invite them to co-review.
GARY: I hope you’ll join me for the Tweet Chat tomorrow to talk about this more, and send me your comments about what you think, or what movement you see on this topic. Keep an eye out for discussions on #PeerReviewWeek21 #IdentityInPeerReview #NPAW2021.
*For an interesting discussion of how this aligns with gender (practical work ending up with women, and “intellectual” labor with men, I recommend taking a look at “The Marie Curie Complex” by Julie Desjardins.