How my month of trying to review a preprint every day went
TL;DR: Not as well as I’d hoped! But also, not as badly as I thought.
In November, I set myself the goal of reviewing a preprint a day, posting the preprint review on PREreview, and then letting the authors know that I’d carried out the review.
In the end, I calculated on my original blogpost that during the 16 regular working days I had available in November, I reviewed 4 preprints:
Blogpost and my review of “Decades of systemic racial disparities in funding rates at the National Science Foundation“
Blogpost and my review of “Knowledge and motivations of training in peer review: an international cross-sectional survey“
Blogpost and my review of “Online training in manuscript peer review: a systematic review“
Blogpost and my review of “American Postdoctoral Salaries Do Not Account For Growing Disparities In Cost Of Living“
Overall, this actually ended up being a 25% rate across the month for days that I was actually in the office. In November, I worked the General Election in the U.S. and went to a conference in Indianapolis, and it also included the Thanksgiving holiday, which reduced my regular working days a lot. After that, I have some excuses: some intensive working days and a dose of the man-flu.
However it was not the “Preprint Review November” that I envisioned, and I have some thoughts.
It’s very lonely, and somewhat intimidating
Reviewing by yourself and under your own incentive requires a lot of motivation; and also, frankly, a lot of confidence. Who am I to take up this articles and send people my unsolicited thoughts? (I mean, aside from being a white man in science).
Everyone that I contacted (which ended up being 3 separate contacts, as two preprints were by the same corresponding author) engaged with me and gave thanks for my thoughts and feedback. That said, everyone could be described as an early career researcher; and while I’m outside academia, I do also recognize the potential power dynamic. That said, I think everyone got the sense that I was coming from a place of genuine interest and curiosity.
But it remains the case that I was not asked to review the preprints originally, and was only under my own pressure to do so, and even with engaging discussions, it felt a little different to being invited to review for a journal.
I actually also carried out a review for a journal during this time. There is something to be said for the request, and outside accountability, that an editor provides. Also the review process I was engaged in was one that very clearly had taken my feedback on board - of course, with the compulsion by the journal to do so. This raises the dark question - is reviewing more fun when I wield some power in the dynamic?
I don’t think so, really, but I think the structure that a community process provides is crucial, and I’ve been thinking about this more in our peer review work with undergrads, and some other experiments in peer review I’ve participated in. As someone who participates in academic work but is outside the ivory tower, strictly speaking, I think this self-led exercise in peer review of preprints felt to me a little bit like yet another way that I’m on the periphery of things (by my own choice), and the limitations that that imposes on having a scholarly community.
Based on these thoughts, I’d be curious how people working at universities feel about engaging in this process in a similar manner to what I just tried. I’d be willing to bet that this isolating dynamic feels similar for early career researchers, or people who are not the “big names” in their fields.
This has motivated me to think more about community efforts to review, or discuss preprints. I’m excited for the potential directions of preprint journal clubs for this reason.
It’s hard to self-motivate
Reviewing for journals is generally enjoyable for me because I get to read interesting work, but the outside compulsion to do so by an editor adds an element of coercion that doesn’t exist with the preprints. I have a pile of preprints and papers that I’d love to read “when I get around to it” and, of course, have not. Reviewing them is a very useful way of reading them, and more so with preprints. I think this is less an issue with reviewing per se, and more to do with time management - I need to re-organize my time to read things generally, and reviewing preprints would make that a useful exercise, not least because I can look up my reviews in the future as a reminder of what I thought about the paper, for future work based on it.
I’m not breaking the network effects yet
Every preprint I reviewed, I heard about through my network - although I will say I did not know any of the authors personally before engaging in their work. I do have searches set up through Europe PMC for new preprints each week; the search criteria are pretty narrow (i.e. I’m getting an update of new preprints about peer review, which is not a sizable body of literature week-to-week!) and perhaps I could expand my criteria, dip into my old developmental biology and ubiquitylation knowledge and start reviewing a bit more widely, based on topic and not on what cool preprints I hear about.
That said - putting a preprint up on social media, a conference or through communities was how I found out about these preprints. So there’s value in working to amplify your work into other people’s channels, rather than relying on people just stumbling across it. Let’s face it - this is how most published papers get cited (which is why I think citations are a rubbish metric to measure scientists).
I need to see how this feels
I’m working on getting some preprints out the door and ask people to critique them openly and widely - and then see what happens.
I haven’t yet experienced someone reviewing a preprint without my solicitation or advertisement, though I’d love it if someone did. I wonder if I’d be annoyed? Who wants to poke me and find out?