When Do Checks Become Review?

Any paper can be uploaded to the web without vetting. But to be deposited to a preprint server, papers usually undergo a minimal set of checks. Few would argue those checks qualify as peer review. But what if the preprint server conducted twenty “rigorous pre-publication checks” like VeriXiv aims to do? Well, even that isn’t being referred to as peer review.

What, exactly, is the minimal amount of assessment necessary to qualify as peer review? It’s a theoretical question, but one with important practical implications. Funding policies often refer to “peer reviewed” scholarly publications and journals get put on the naughty list for lesser-than levels of peer review. But the line of demarcation between “peer reviewed” and “not peer reviewed” seems to be growing more hazy as the number of states between these two increases. Or is it?

This week, I’ve spent a little time with the Sorites Paradox Wikipedia page, a concept I vaguely remember from undergraduate philosophy courses. This paradox is usually illustrated with a heap of sand from which individual grains are removed. Removal of a single grain does not cause a heap to become a non-heap.

The paradox comes from considering “what happens when the process is repeated enough times that only one grain remains: is it still a heap? If not, when did it change from a heap to a non-heap?”

Something additional will need to be added to the pile of twenty checks that papers will undergo on VeriXiv in order for the whole process to amount to a full Peer Review heap. But just how many more things are necessary?

Maybe such preprint experiments are too nascent to base a reinterpretation of established concepts upon. Perhaps it would put us on more solid footing to highlight examples from our more commonly-held understanding of scholarly publishing, in order to triangulate a precise measurement of peer review.

To that end, let’s take these three more points under consideration:

• papers do not need to change as a result of peer review (ex. paper is accepted without changes, or editor accepts author’s refusal of suggestions),

• the peer reviewed Version of Record does not necessarily mean Final Version, given that papers may be corrected at any time, and

• a peer reviewed paper may be retracted, but it cannot be unsaid that the paper was peer reviewed.

A theme that I take from these bullets is that peer review does not have a necessary relationship between time, venue, version, or modifications. Long story short, the subject seems complicated, if we want it to be. Maybe this is all just another case of AJ overthinking it.

Maybe one of the proposed resolutions to the Sorites Paradox offers a way out here, such as the “group consensus” reply. That likely will be the case, given that all of these conventions are socially-constructed anyway. NISO has developed a pretty thorough Standard Terminology for Peer Review document that is effective at defining the nomenclature.

What we still lack is a broad consensus of what exact activities amount to peer review. Perhaps not outlining such a standard is the best possible solution, given variations in sub-disciplines. Personally, I would be okay calling it peer review if only one individual vetted a paper along one dimension—so long as it was clear to anyone who came across the paper that that was the case.

This fits in with my broader philosophy of open science, where, it’s less important that a p-value is “significant” or not, but rather that the data that went into making that judgement is open for scrutiny or not.

Idk. I’m just some guy. What do you think?

***************************

Like this content?

Consider supporting: https://buymeacoffee.com/arthur.boston

Or leaving feedback: https://forms.gle/eDiiYfeEMNciYdJr5