Knowledge and motivations of training in peer review: an international cross-sectional survey: Review for Preprint Review November

November 4

Today’s reviewed Preprint is:

Title: Knowledge and motivations of training in peer review: an international cross-sectional survey

Server: medRxiv

Link to preprint: https://www.medrxiv.org/content/10.1101/2022.09.03.22279564v1

How I found out about it: Presentation at Peer Review Congress by first author

Time taken: 3 hours

Link to my review: Review on PREreview

Contact with authors:

Sent an email to all the corresponding author after the conference, stating intent to review this and another of their preprints. I will email them after I have reviewed their other preprint and posted that review.

General comments:

D’oh

My first comment is that, at time of writing on the 4th, I have placed the review for this preprint on the wrong preprint on PREreview, and am asking for PREreview’s help about what to do next….sigh.

11/7 update: PREreview have fixed my error; the review is now on the correct preprint.

Overview of the Preprint and my process

I took a very long time with this, because I had a lot of comments that arose regarding the limitations of the population surveyed, and it took me a long time to write out (and then self-edit, to make it attempt to make it more concise). It’s a good set of results, but I think the description of the population as representative of biomedical researchers is not strictly correct, and indeed undersells the results to me, because they took a very systematic approach to surveying corresponding authors about their attitudes to peer review and training. This is very neat, because this is likely to select for a very priviliged group in academia, and getting their opinions on these matters (and I was happy to see how well they matched up to some of our work on early career researcher attitudes) is very important. And specifically this targeted a group who also reported being involved at journals at a high rate, so this also captures some of what they know about what their journals do, and maybe give a sense of the thinking of some of the people behind journals.

Some thoughts on self-citation in review

I am not sure how I would review my review. First of all, I think it’s maybe too wordy, but I tried to structure it to be as useful as possible.

But also, let me be honest about a journey I went through reading it.

I noticed that a lot of results were aligned with some of the things we’ve found in our other work, or gave new insights to our work, such as the survey of early career researchers. When I looked up the citations of some of these things - I saw we weren’t cited anywhere.

Was I annoyed? I’ll be honest - a little, yes.

But reflecting on this - why am I annoyed? Why should I expect that someone has cited my work? I think in academia we have this constant perception that all scientists and the academic system are objective, and of course if your work is relevant, it should and will be cited. But this is based on a flawed assumption - that everyone finds the relevant work to cite, and then cites it, through some objective mechanism. And we don’t - we cite what we’ve heard of, what we know, or frankly through a search engine (and not a proper search a librarian helped you with). And sometimes we deliberately (or subconsciously) don’t cite things written by certain people. We have this nonsense mentality that “our data speaks for itself”, and that citations are meritocratic and free of bias and subjectivity.

To illustrate this point: the last author on the preprint is someone who, turns out, has done a lot of work on peer review. And is definitely more well known than me in the field, by far.

I went back through my papers. Have I ever cited their peer review work in my work?

Turns out, I have not.

So. It’s important to remember that citations happen because of network effects, and who we know, or know of. And it makes complete sense that the two sets of authors haven’t come across each other’s work, because there’s slightly different disciplinary foci (which I bring up a little in the review, now I’m tuned into it).

Here’s the tricky part, which you’ll see in my review.

Do I cite myself in the review? Do I share these papers that I think are relevant?

In the end, I opted not to self-censor - and to share them in the text. The whole point of this exercise for November is to go through and reflect on what open peer-reviewing is like, and lay out my thoughts around it, however ugly they are!

Here is the issue I struggled with in my mind:

I think they do not know of this work; I think they might genuinely find it interesting, or useful. But dumping your work into a review (and of course reviewers may explicitly say “you should cite this” in their review) is not a great look. But, at the same time, if it’s relevant to the work - why not? So the reasons I came up with for why to put my papers in:

• I’m interested in review as a way of discussing scholarship, and so of course if I’ve written other things that are relevant, it makes sense to bring them in.

• I also, as a consultant, don’t really (well, a little, I am human) care about being cited. But I am definitely not incentivized to boost my citations by dropping all my articles into a review and getting the author to cite them - no-one looks at my citations, so it’s really only my ego that’s affected.

• I do want my work to be useful to others. This is a group of people who work on similar things, so this seems a good opportunity to share the articles. And bear in mind - I could (and let’s come clean - I did) email someone privately with the papers. So I should do it publicly if I’m doing it at all.

• Furthermore, as this is a public review, and if people come along and look at the preprint, it’s maybe useful for them to see other related things too in the review - this is part of why I discuss the articles in terms of the context of why this new work adds further insight, or shines light on gaps, in my work. Again, I think peer review should be centered around scholarly conversation, and this is a work in progress, so discussing my work in the context of their work, and my thoughts about it (I spent 3 hours looking at this preprint and thinking, so I did really think) should hopefully be a legitimate part of scholarly discourse.

• This review is not private, and isn’t going through an editor, so there isn’t a power dynamic where I could be perceived as trying to use the editorial process to force an author to cite me as part of getting into the journal.

• Yeah, OK, I wanted to show off the work I did. Sure! I did all this work on peer review, I’m proud of it! Anyone who meets me at a party now is bored to tears with me talking at them about undergrads in peer review, I bore for Ireland in person (I could give the uncle in Derry Girls a run for his money), so I’ll do it in writing too. I am not an automaton, I am a flawed human, so at least I can try to be honest about it, even if I think there are also a lot of valid reasons I could hide behind.

What do you think? AITA?

Please feel free to comment below.

Any updates will be posted below here with a date!