News and comment
On Peer Review
… a limited tool at best – never a proof
Many people may have a mental image of peer review as a process where white-coated scientists re-run the experiment in laboratories or repeat the research out in the real world.
After this they have long meetings in board rooms to discuss the paper and, finally, they confirm that the research and results described in the paper are rock solid and beyond doubt. They then approve the paper for publication in a journal.
The belief in peer review as proof of scientific fact becomes conflated with concepts like truth, beyond doubt, trustworthy, reliable, beyond dispute, the gold standard of science and so on. Sadly, for people who hold these beliefs, peer review is nothing of the sort.
So, What is peer review?
Peer review is a step in the publishing process where an editor/publisher attempts to weed out papers that are spurious or obviously in error. It is a process where papers are vetted to ensure they present a cogent argument based on recognized scientific analysis.
At a superficial level peer review can be as simple as a spelling or syntax check, that scientific terms are correctly used and that the paper reads okay. This might sound lightweight for a journal at the forefront of scientific knowledge but presentation is important.
This part of the process can also include checks on the visual aids used in the paper such as photographs, diagrams, graphs and tables. Are these clear and understandable? Do they support the points made in the body of the paper? Are they relevant or do they just look good?
At a deeper level the peer review process addresses issues like: ‘Is the hypothesis sound and relevant and is it supported by the Introduction?’ Is the logic of the paper sound? Are the methods sound? Do they relate to the hypothesis? Is the argument well constructed, brief and to the point? Is there anything missing in the chain of logic? Does the paper present new information or does it support existing knowledge?
Peer reviewers are appointed by the editor and most scientific journals are quite specialized as to the field of science upon which they are writing, so reviewers need to confirm whether the paper satisfies the scape of the journal.
Given that there are many different journals and editors, it becomes likely across the spectrum that there can be different standards and requirements as to what the editors and publishers require. It is fair to say the term ‘peer review’ has taken on a life of its own, free from real meaning and certainly not a re-running of an experiment.
Not a proof
Peer review is not intended as proof and yet peer review is trotted out all the time as the gold standard of scientific proof.
The ‘proof’ or disproof of the findings of the paper comes when it is put under the blowtorch of criticism of the wider scientific community and, most importantly, when it is put under the blowtorch of Test by Replication.
Peer reviewers are not claiming they have done the experiment again or made the same observations or done the same maths and obtained the same results. They are not claiming to have replicated the research. This is a really important point.
Behind closed doors
There can be an area of grey when it comes to the transparency of the peer review process because the peer reviewers can choose to do their work anonymously. This conflicts with a characteristic of the scientific method which requires that science be open.
It is true that this anonymity aspect is not always the case and reviewers can choose to be anonymous or open. Despite this discretion, it is common for peer the peer review process to be dome behind closed doors. Where this happens, the peer reviewers are simply put in a position where their background, track record, expertise and even their strongly held opinions cannot openly be taken into account. The peer reviewer may simply be prejudiced against the thesis of the proponent and, by exercising this prejudice discretely and anonymously, can stymie the publication of a paper that is otherwise a valuable contribution.
Usually a paper would have two or three peer reviewers and the writer – the proponent – has the right to contest the comments coming back to the editor from the reviewers. But where the reviewers remain anonymous, it just makes an open scientific discussion harder.
Conflating peer review with proof – an example
An example of the conflation of peer review with proof came with an ABC Television Media Watch segment some years ago. Not only were the two concepts conflated, but the high profile Reef scientist, Ove Hoegh-Guldberg, Ph.D., Professor of Marine Science, University of Queensland, gave a ludicrous analogy of the credibility of peer review.
Hoegh-Guldberg was quoted as asserting that the idea that reef science can’t be trusted because it’s only peer reviewed, was, ‘… just ridiculous.’
Hoegh-Guldberg was then cited as saying that peer review was, ‘… the same process we use when we’re studying aeronautics, which produces planes that we travel on …’
On first hearing this it sounds like a compelling analogy. But think it through. There is no way an aeronautics engineer would accept something say, a new alloy for inclusion in an aircraft design, on the basis of peer review alone. The engineers would go for the higher standard of replicability. The alloy would be tested to destruction many times in a process of replication over and again to see if it stood up to the claims made about it.
The engineers would depend on three things; replication, replication, replication.
Replication, not peer review, would be the key to the proof.
Weeding out fraud, spoofing, dumb errors
In its quality control aspect, peer review might be able to weed out totally spurious papers, rants by complete cranks, fraudulent works, mathematical incompetents, mischievous papers by tricksters and even April Fools’ Day jokes. It is not, however, even guaranteed to perform that role very well.
Vulnerable to spoofing
A recent hoax involving peer review was first reported in ‘The Times’ newspaper in the UK and subsequently reprinted in ‘The Australian’ newspaper on 9th January, 2019. The journalist, Rhys Blakely, is The Times science correspondent and the article in The Australian is headlined, ’Academic faces sack over hoax that fooled academic journals.’ The article outlined the plight of Peter Broghossian, an Associate Professor of Philosophy at Portland State University in Oregon, USA, who is facing the university’s censure over his role in the well-intentioned hoax.
Led by Broghossian, several academic wags spoofed the peer review process and wrote 20 spurious papers, all of which had the trappings of serious scientific papers. They submitted them to academic journals for publication. The papers were meaningless rubbish but that is not the way the peer reviewers saw it. Of the 20 bogus papers put out to peer review, seven were accepted for publication – that is 35% of the total!
Dr. Broghossian and his colleagues were shocked by the ease with which the papers were accepted. Although it may have been a hoax, it confirmed peer review is not the robust gatekeeper of truth that many people believe.
Another example
This involves papers written by research student, Oona Lonnstedt, who conducted research at James Cook University and gained a Ph.D. at the Australian Research Council Centre of Excellence for Coral Reef Studies. Lonnstedt then went back to Sweden where she did the further research which has tripped her up.
The paper which set the suspicions running was at Uppsala, Sweden. It was about the effect on small fish of ingesting ocean micro plastics and how this affected the ability of the fish to grow, hunt and survive. The paper was published in the high profile journal ‘Science’ in 2016, and was challenged by two concerned scientists within a week of publication. The challenge came after the paper had been peer review.
Lonnstedt’s paper has been examined by Uppsala University and been retracted. The report of the University’s Board for Investigation of Misconduct in Research was published in December, 2017, and found that Lonnstedt had fabricated data. Both Lonnstedt and her supervisor were found to have engaged in research malpractice.
Another body in Sweden, The Central Ethical Review Board, found that Lonnstedt and her supervisor had committed scientific dishonesty.
Again, it is noteworthy that the peer review process did not detect the issue of scientific dishonesty in this case.
Resplandy et al, 2018
The case of Resplandy et al, 2018, also illustrates the unreliability of peer review. This paper was published in the prestigious journal ‘Nature’ in 2018. (Resplandy et al, Quantification of ocean heat uptake from changes in atmospheric O2 and CO2 composition).
Upon publication it soon became evident to certain readers that there was fundamental flaw in the analysis. The flaw multiplied the degree of uncertainty on the paper so much that, even after a correction had been made, the publishers eventually decided that the paper could not be allowed to stand and was retracted.
It is interesting that the first reader to raise the alarm was not a marine scientist, nor a climate scientist, nor a person with a Ph.D. Rather, he was a analyst in the finance industry and this shows that one should not be intimidated by a host of ‘experts’ who have peer reviewed a paper.
Retraction Watch
Retractions of peer reviewed are not rare and I invite you to visit the Retraction Watch website https://retractionwatch.com which commenced in 2010. At the start, the founders knew that retractions were happening on a regular basis but wondered if there would be enough to sustain a website. They thought they might have been able to identify around 80 cases in the first year but, in the event, they found over two hundred.
The rate for Retraction Watch has continued unabated. As of January, 2020, the site has reported on 21,792 retractions. All of these had been peer reviewed prior to publication.
Conclusion
The belief in peer review as proof of scientific fact is false. The flip side of this belief – that the lack of peer review shows a paper is untrue – is also false.
Peer review is not a proof of anything and is not intended to be. It is vulnerable to fraud, hoaxes, spoofing and simple errors of maths. Peer review is not a replication of the original experiment or research.