“Preprint Review November”: Taking some time to experiment with preprint peer review

“Preprint Review November” was the catchiest title I could come up with for my plan to attempt to review a preprint per day(ish) during the month of November, which I’m setting up as a wee experiment in how and why to do preprint peer review.

Preprints are drafts of research articles launched into the world by their research teams, in various guises and with various intentions. Some (most, in the biological sciences?) are archived at the same time as being submitted for curation at a magazine, but others are being submitted with the preprint being the final destination (which is more common in, say, the physical sciences) or as a very clear work-in-progress (working papers, such as in the social sciences).

I’m not really interested in the magazine-curation game (or journal publishing) anymore. You have to pay to play, and also pay to read. And increasingly, as I study peer review in greater depth and after attending the Peer Review Congress recently in Chicago (more soon on that), I am thoroughly underwhelmed at the quality of the peer review service that journals undertake, which seems largely to be based on editors sending papers out to their conference drinking-buddies, who are male professors at Harvard and Stanford.

I think we could be doing the peer review service we already do for free ourselves, but I also don’t believe we can just switch to that system without appreciating the real difficulties and biases that come with it first. So, for the next month, I’m going to not only commit myself to reviewing a preprint a day, but also recording various pieces of data around it, as honestly as possible.

For example: I really love reviewing, and I really enjoy reviewing preprints, but I find it really hard to get myself to do it. I have a few preprints I promised I would review from months ago - somehow the always falls to the bottom of the pile. Can I find what will click to get me actively reviewing on a regular basis?

Another question: how do I seek out preprints to review? Am I going to end up reviewing my friends’ work, and asking them to review mine? And with so much concern on Conflict of Interest - does this matter, especially if we all do this openly, if at the end of the day the critiques are valid?

I’m hoping to come up with some more concrete observations to revamp my “Preprint Peer Reviewer Toolkit”. I’m hoping that I can set up a workflow where I am reading a preprint daily that helps me with my own work and research, that I can then also provide feedback on (and reading it with a goal to provide feedback will, I hope, make the reading to the preprint more useful to me too). Perhaps I will be able to get a sense of whether the feedback is useful to others - and honestly, whether that even matters, and the worst thing is I have a better system for reading and appreciating research, that also now has some of my thoughts openly presented to the world for whatever use they may be to anyone who comes across them.

Ultimately, I want to get at why I am finding this so hard to do at the moment. I’m so intrigued as to why someone as highly motivated as I am to be reviewing preprints finds it so difficult to do. As I go through the process and reflect on various factors about the process, hopefully I can come up with some answers - and maybe come out the other end with this being an essential daily activity.

If anyone reading this has any ideas on data that would be useful to collect or factors I should be thinking about - please drop me a line or comment below! I hope this can be useful to other people’s work.

As the month goes on, I will blog about the preprints I review, and drop the links to the blogs, and their corresponding review on PREreview (my current preferred site for depositing my reviews) in a list here:

11/3: Blogpost and my review of “Decades of systemic racial disparities in funding rates at the National Science Foundation“

11/4: Blogpost and my review of “Knowledge and motivations of training in peer review: an international cross-sectional survey“

11/7: Blogpost and my review of “Online training in manuscript peer review: a systematic review“

[11/8-11/11: Limited posts, as I’m working at my local polling station for the U.S. General Election, then headed to Indianapolis for the NABT Conference.]

11/14: Blogpost and my review of “American Postdoctoral Salaries Do Not Account For Growing Disparities In Cost Of Living“

11/15-11/18: I was overseeing a data collection effort for one of my projects at a remote site - and ended up not having the time to set aside to review preprints these days.

11/21-11/23: I was engaged in peer review of a journal article, and in the lead up to Thanksgiving.

11/24-11/25: Thanksgiving Holiday!

11/28-11/30: I was ill with what was most likely a cold (based on testing) that was not very serious but rendered me feeling capable of mostly doing admin/other non-cognitive-heavy tasks.

12/13: Reflection: In summary: over the course of my 16 working days in November, I reviewed 4 preprints. ¯\_(ツ)_/¯

See my summary post for final thoughts!