The post-publication peer review (PPPR) has become a heated discussion topic. Precisely, the issue is not whether PPPR is necessary (the opposition comes only from the most system-entrenched dinosaurs or from those who have their good reasons to fear PPPR). It is the anonymity of PPPR which is under debate. The discussion was kicked-off by Michael Blatt, botany professor at the University of Glasgow, in an editorial in the journal Plant Physiology, where he is editor-in-chief. Blatt’s concern was the perceived lack of responsibility behind the anonymous public criticisms left on the PPPR internet platform PubPeer. Soon enough, his editorial was hotly debated on PubPeer as well, including by myself, and most prominently, by the physics professor from University of Nottingham, Philip Moriarty. Moriarty has a strong record of promoting research integrity and reproducibility, yet he always does it in the open and is strongly opposed to anonymity in academic discussions. Hence, he lent his support to Blatt by calling for openness at PPPR, decrying the insistence on anonymity behind even the most “innocuous” PubPeer comments.
I have been in personal contact with Blatt and Moriarty, and my initial scepticism gave way to my support of their ideas. Importantly, they both agree on the need for the whistle-blower protection where data integrity concerns are reported. Blatt is convinced that the identity of the whistle-blower should be known at least to the direct recipient of evidence, whose task would be to treat it fully confidentially. Moriarty’s case however is less with the accusations of possible data manipulation and misconduct, but with the scientific discussion, which is exactly what PPPR is about. He sees no reason to hide one’s identity while expressing objective and thought-through criticisms of a published paper. I tend to agree with him, and will provide below examples on how anonymity in PPPR can become counter-productive or even toxic, while signed PPPR can lead to remarkable results and to personal recognition for named scientists involved therein.
PubPeer was intended and still describes itself as “The Online Journal Club”. Its core concept is anonymity; commenters are free to post either as unnamed “Peer” or as “Unregistered Submission”. In the latter case, not even PubPeer operators have any clue of the commenter’s identity (i.e., beyond the computer IP address). This is exactly why PubPeer never became a proper “journal club” it strived to be, but a whistle-blowing platform instead. As such, PubPeer achieved tremendous results. The list of publications, which were corrected and even retracted following the anonymously presented evidence for data manipulation (mostly image irregularities such as duplications, splicing or inappropriate Photoshop-ping), is very long. Major journals like Nature and Science had to retract papers, while even the most influential scientists had to face institutional investigations or have lost their jobs, thanks to the activities of the anonymous PubPeer users. Meanwhile, many journals now actively track PubPeer and take notice whenever another one of their papers is flagged for possible data manipulation.
Yet where true academic PPPR is concerned, PubPeer’s default anonymity seems more of a hindrance than advantage.
Until their public coming-out on August 31st, even the PubPeer operators themselves were anonymous. But even though their names are now known, the people behind PubPeer for some reason still prefer being seen as an anonymous collective. Their communications are rarely signed by a name, with rare exceptions as in a recent Vox article on peer review shortcomings. There, the PubPeer President and tenured CNRS researcher Brendon Stell was quoted promoting PPPR. Elsewhere, the letters which PubPeer exchanged with Moriarty and which were then published in Times Higher Education, were signed as “The PubPeer organisers”. Fittingly, the topic of their discussion was the anonymity of PPPR. The PubPeer organisers wrote to Moriarty:
“Our experience has been that anonymity greatly facilitates scientific discussion with little if any effect on the quality of comments. […]
The main arguments against anonymity are that it allows commenters to blacken the names of their competitors with impunity and to hide conflicts of interest. Again, our practical experience has been that these issues rarely arise, for several reasons.”
Yet, I myself witnessed on PubPeer authors defending their own papers by anonymous sock-puppetting. There, PubPeer sometimes interfered to delete such comments, but other suspicious posts were left standing. Also the following example shows that anonymity of PubPeer can be easily abused by less trustworthy commenters to hide their ulterior motives and conflicts of interest (COI).
A seemingly academic discussion started some time ago on the PubPeer thread for a high-profile publication in Cell by Yosef Buganim, currently group leader at the Hebrew University in Jerusalem. This 2012 paper dealt with pluripotent stem cell reprogramming and came from one of the most influential labs in this field, namely that of Rudolph Jaenisch at Whitehead Institute in MIT.
An “Unregistered Submission” and later a “Peer 1” questioned the findings of that publication, citing dissonance with research published elsewhere, including in other papers from Jaenisch lab. Below some of the relevant comments, now moderated by PubPeer:
Buganim replied to the criticisms, by posting as Peer 3 and signing with his name. He was supported by anonymous commenters (Peer 4 and Peer 6). The academic argument over data interpretation quickly escalated, with Peer 1 being personally attacked by Buganim and others.
After I questioned these tactics with my own comment, Peer 1 was soon revealed to be none other than Jacob Hanna, another Israeli stem cell researcher whose career is closely intertwined with Buganim’s. Hanna’s postdoctoral studies at the Jaenisch lab overlapped with those of Buganim, the two were therefore direct former lab colleagues and currently direct competitors for limited stem cell research funding in their small country, Israel. But this is only one issue.
Hanna, who leads a large lab at the elite Weizmann Institute of Science, has been accused of heavy data manipulations in a number of his own publications, which include his PhD studies with Ofer Mandelboim at the Hebrew University, his postdoctoral research with Jaenisch at MIT and papers from his own lab back in Israel. Hanna had to retract one publication in the Journal of Clinical Investigation and issue corrections for 10 of his papers due to image duplications and other data irregularities. When faced with accusations of misconduct in regard to his paper in Blood, Hanna blamed his “medical trainees”. And if this is not enough, Hanna apparently somehow managed to have most image evidence of data irregularities in his many criticized papers disappear from PubPeer. His research record did not remain untainted for long, since PubPeer quickly restored most of those images from their own backup server.
As soon as Hanna’s anonymity was blown, and PubPeer was informed, they swiftly interfered. The majority of comments in that thread was deleted in one go, including all revealing references to Hanna. Yet PubPeer let most of his (now once again fully anonymous) criticisms of Buganim’s work stand.
Given all this information, can Hanna be ever deemed as unbiased reviewer for the papers of his former colleagues and direct competitors? For all we know, he might even (rightly or not) suspect some of them to have anonymously flagged his own data integrity issues in the first place. Given the bombastic COI, how reliable are Hanna’s criticisms of the Buganim paper, especially if they are issued from under cover of anonymity? Also, who are these Peers 4 and 6 who took Buganim’s side? Can we trust them to be unbiased observers?
Hanna did not reply to my contact attempts through Twitter and his institutional email. I then approached Buganim about confirming Hanna’s identity as Peer 1. Buganim did reply, but was reluctant to get involved and indicated the conflict was settled for him. However, his replies made it clear that Buganim is likely to have solid evidence that his colleague Hanna has been that anonymous commenter.
The main criticism of traditional journal peer review is its intransparency. COI are often suspected when fishy manuscripts are waved through towards publication, while good research studies are sabotaged by competing or even personally hostile colleagues acting as peer reviewers. Of course, the job of a good journal editor is to prevent such COI, yet the very system of peer review intransparency allows many editors to rig the game without anyone noticing.
The rotten scientific literature coming out of this rotten peer review is what PubPeer wants to combat with their model of anonymous PPPR. However, the above example shows that even the most outrageous COIs can easily creep into anonymous PPPR, and the moderating authority either tolerates or is not always sufficiently able to prevent them.
This is only one argument for the PPPR to be open and named. As the following example shows, signed PPPR is much more likely to be taken seriously by authors, journals and relevant research institutions. Eventually, scientists engaging in open, signed PPPR can even advance their careers and make an internationally recognised name for themselves.
Open protest gets noticed
Johannes M. Dijkstra is immunologist and lecturer at the Fujita Health University in Nagoya, Japan. Though himself not a member of the mile-high impact factor club, Dijkstra has a strong record of named and open PPPR, where he challenged publications in biggest elite journals like Nature, Cancer Cell, Cell as well as in Science. In his email to me, he described his approach to open and signed PPPR:
“I only discuss articles “close to home” when it is absolutely necessary to keep my research field clean, and otherwise choose articles that are further away career-wise and emotionally. I engage in PPPR because I feel that we scientists ourselves have to do something to reduce the flood of bad papers, or silly messages within otherwise good papers, and that starting fair discussions is the best way to do that.”
“I am very lucky to have a boss who doesn’t consider PPPR a waste of time, but understands that in regard to science as a whole this was a very efficient usage of my time. There is nothing which spoils so much time, thinking and money as bad articles, especially in top journals. I didn’t experience any negative career or networking impact so far. […] You have to be very right if you publicly challenge the articles of others, otherwise people will only consider you as a nuisance. […] If I criticize people to such a large extent, they have the right to know who I am and maybe hold me accountable to the same standards”.
Recently, Dijkstra turned to PubPeer to express his criticisms and even his suspicions of data manipulation on another recent Science publication, Okoye et al, 2015, from the lab of Philip Ashton-Rickardt, immunology professor at Imperial College London. Initially, Dijkstra’s detailed criticisms of that paper were directed at the journal in confidentiality and led only to a controversial Correction, where apparently manipulated data has been replaced with new one. Seeing this, Dijkstra decided to issue his open and signed peer review on PubPeer. From there, things began to move properly.
The correction was soon followed by an editorial Expression of Concern, prompted by the institutional investigation from the side of the Imperial College. Though Dijkstra was supported by anonymous commenters on PubPeer when flagging additional image manipulations, the question remains if his concerns would have been taken seriously if he himself chose to comment anonymously as well. This is one discovery of what the Ashton-Rickardt paper contained, as presented on PubPeer:
There are quite a number of other examples where serious image manipulations were anonymously flagged on PubPeer, only to be “fixed” with some embarrassing corrections or painstakingly ignored by the authors and the journals. There is probably an even higher (dark) number of cases where signed concerns were confidentially mailed to the journal editors, and then quietly disappeared in their drawers for all eternity.
Combining the advantages of both approaches might actually do the trick in making reluctant editors sit up and take notice, and this is exactly what Dijkstra did in the case of Okoye et al. He felt compelled to take “the public route through PubPeer”, because Science editors initially responded to his concerns “inadequately”:
“I hope that Science will change its procedure in dealing with suspicions
of misconduct. If they would have investigated the matter more seriously after my initial complaints, we would never have arrived in the current situation. Then there would have been a rather clean retraction, and everyone could have gone back to business. They combined a poor attitude (to avoid retraction at all costs) with lack of proper procedure and a lack of skill to “feel” the severity of the flaws in the Okoye et al. article”.
Whether this paper will be ever retracted by Science, is another question. Important for now is, the academic community has been made aware of its flaws and can now make its own conclusions about its real value. Signed PPPR was an absolute success story in this case. Its author Dijkstra has shared in his email to me his advice for all those who wish to engage in PPPR:
“I think it is better for science and more honorable to make comments under one’s own name, but that I do understand a number of reasons for which people can choose not to do that. I can understand that people want to be able to discuss science on PubPeer or other social media without having to polish their words, or without full engagement of their “science identity”. I also have come to kind of enjoy it as pleasantly tranquil, this lack of names on PubPeer, as it keeps the focus on the issue and not on “my God, who’s saying this”. But then there are instances in which I strongly think “grow some …..” if I see people anonymously attacking other scientists.
In the larger picture of things, I think that the anonymous character of many of the complaints is bad for achieving results. The anonymity makes “us” just look like a bunch of mice shouting over picture duplications. If we really want to change things, we have to sit at the table with the other parties, and that means showing our face. I hope that PPPR becomes more common and more normal. It is the best and maybe only way that we scientists can try to improve the standard of publications, instead of always pointing at others who should do that for us”.
However, also Dijkstra is fully aware of the risks, especially those untenured scientists face at named PPPR. Also he has once warned a colleague without permanent position against publicly criticising a certain problematic paper. Dijkstra was concerned about the possible dangers to the career such “troublemaking” might have brought. The resulting conflicts with authors might be strenuous, while potential future employers might not at all value such activities.
The decision about signing one’s name under a public post-publication peer review is therefore a personal one. Sometimes it may be indeed wiser to comment anonymously, but even then one should always be objective, stick to the facts and basic ethics, and avoid acting on hidden conflicts of interest. It is very tempting to leave less thought-through criticisms under the conditions of anonymity, simply because it is so easy to separate oneself from them. This argument was even used by an anonymous commenter in the PubPeer discussion with Moriarty. However, even anonymous commenters should apply same considerations to the content of their criticisms as if they were signing them with their names. Imagine, it is your own paper someone wishes to anonymously comment upon.
Before signing the comment, every scientist doing PPPR has to weight his or her risks, but also consider if these risks are real or only perceived. Making a name in your field as a competent and incorruptible scientist might get you noticed also by the right people who will value your work.
Participating in open and signed PPPR may not be so bad for your career after all.