Lawyering-up Research integrity

Spiderman’s lawyer is having you for dinner tonight

Spider researcher Jonathan Pruitt is accused by his coauthors of data manipulations, after 3 retractions they demand more. A lawyer's letter was supposed to stop that, but Pruitt tells me: "I'm happy for folks to engage in public discourse about my data integrity""

The Canadian shooting star of behavioural ecology Jonathan Pruitt used to be everyone’s best friend. Up until his co-authors accused the spider researcher of data manipulation and achieved retractions for 3 of his papers. His reputation in tatters, his research record facing even more retractions, his elite funding as one of Canada’s “150 Research Chairs” and prestigious job at McMaster University in danger, Pruitt did what every red-blooded academic in his position does: he deployed lawyers against his critics.

As Science reported, the lawyer’s letter was so scary that some journal editors panicked and hid under their tables, waving a white flag of surrender. No more Pruitt retractions unless specifically requested by the ongoing investigation at McMaster, no matter what the evidence submitted by co-authors proves.

The original recipients however, all of them former Pruitt co-authors, are not complying, they continue vetting Pruitt’s publications for further irregularities. As I learned, nobody signed anything or announced to adhere to any of lawyer’s demands. And why should they be afraid, the lawyer’s letter is not even really threatening anyone. It is a cack-handed attempt in intimidation which even Pruitt now apparently distances himself from.

All the letter from Millard and Company does, is to educate its recipient that nobody is supposed to be scrutinising Pruitt’s papers but his employer McMaster University, and that every retraction must be agreed by the Committee on Publication Ethics (COPE, a scholarly publisher lobbyist), whose advisory guidelines to its member journals are presented by Pruitt’s lawyers as a kind of a law for breaking of which you might end up in prison.

The letter is labelled “PRIVATE & CONFIDENTIAL WITHOUT PREJUDICE”, which is exactly why I publish it here in full, below. I hope the Millard lawyers are happy with this arrangement.

The Pruitt case made international news, it even got its own hashtag n Twitter: #PruittData. The communal effort to uncover data irregularities began with Kate Laskowski, now assistant professor at UC Davis in USA. Laskowski told the story on her blog, and it might explain the success recipe of Pruitt: when he met some scientist with an unproven pet theory, he provided them with the perfect experimental data. Which now proved to have been too good to be true.

In science, many beautifully designed theories collapse on experimental reality. Biology is very messy, complicated and rarely does as told, even and especially when the theory which biology is expected to adhere to is clever, beautiful and elegant, of the kind the editors in elite journals love and the research funders cherish. Which is exactly why so many scientists resort to research fraud, to force scientific reality to adhere to their theories.

Laskowski, then a PhD student, had a theory about animal behaviour in social context. She met at a conference the young shooting star Pruitt, who works with social spiders. Pruitt generously offered his collaboration, and soon perfect data arrived which perfectly proved Laskowski’s theory right. Pruitt’s datasets served as template for 3 papers in respectable journals, until someone contacted Laskowski with concerns. That colleague was Niels Dingemanse, professor at the LMU Munich in Germany, and he noticed there were statistically too many duplicate numbers in the paper’s publicly available original data file.

Pruitt told Laskowski he measured 4 spiders simultaneously, hence identical values. But how can spider act so synchronously and into 1/100 of a second?

Thus alerted, Laskowski started to go through Pruitt’s Excel files, and indeed found many repetitive numbers. Pruitt was quick with an explanation, which sounded a bit weird, but it kind of sufficed to consider a correction. But then Laskowski analysed the files in more details and found entire strings of numbers repeating themselves, with the only logical explanation that they were made up. Once Laskowski removed all repetitive numbers, the remaining ones showed no significant difference in spider behaviour between experimental treatments. Her theory collapsed, so Laskowski had her own three papers retracted, one by one.

Laskowski: “Each line is an individual spider. What I noticed is that the last two pre-treatment values, seemed to be identical to the first two post-treatment values in many spiders in this treatment. I colored those values red here. I also noticed that many values were very similar, but off by one digit in the hundreds column, e.g. 77.65 became 477.56. Those values I colored in yellow.”

In another blogpost on the topic, Dan Bolnick, editor of American Naturalist and professor at University of Connecticut, explains that the journals actually very much can be expected to take responsibility and to perform data integrity investigations, which his journal did. Bolnick even used to curate a publicly viewable (but not editable) Google Docs file to track the reliability of Pruitt co-authored papers. Some papers actually proved to contain no evidence of irregularities. The list was removed now, after Pruitt lawyered-up. Bolnick’s commendable behaviour stands in stark contrast to certain other editors who now hide behind Pruitt’s lawyer’s letter and the McMaster investigation.

Laskowski: “as opposed to being formatted in columns as on Sheet 1, these values were transposed – so that the columns became rows. […] Here on Sheet 2, the duplicate sequences were not just sequences of 4 numbers (as they were on Sheet 1), but rather I could now see that in fact there were whole blocks of duplicated sequences.”

Unlike his lawyer claims, Pruitt even agreed to the first retraction, which happened in American Naturalist. Laskowski quotes him with: “it is well that the paper is being retracted”. But eventually Pruitt changed his mind. Things got out of his control namely. After that retraction, other Pruitt coauthors reached out to Laskowski, the search for data manipulations became a communal effort. Many went on to publish their concerns on PubPeer, where other Pruitt’s coauthors first defended Pruitt, until some switched sides and joined the investigative group effort, like the associate professor Noa Pinter-Wollman at UC Los Angeles. Did these scientists also discover their own beautiful theories, once verified by Pruitt’s data, now in lying in tatters? Possibly.

One former Pruitt collaborator explained why his co-authors initially defended the accused:

It’s worth remember that lots of us considered ourselves to be good to very close friends with him. You just don’t want to believe something like this.

And then they looked into their own papers with Pruitt. A co-author shared this about the raw data behind a publication:

literally 1/2 the data were sequence repeats 10-14 units long but sometimes with select manipulation

That source also described the communal #PruittData investigation:

There are 28 channels, on the slack channel where people are looking at this, each of which is dedicated to a paper. […] The sequence repeat pops up a ton. Formulas are pretty common. One person found that a formula was used but it was like + 20.3 or something to better hide the pattern.

Rows K-O in the Pruitt raw data file “Black_Widow_Data” list independent experimental values for “Darkness”. And yet, values in row O are simply values in M +1.5. The author sloppily even left inside the formula with which the values were fabricated.

Pruitt’s manipulative approach over his former friends does not work anymore. If anything, people are afraid of his revenge now, rightly or not, time will show. As a source wrote to me:

Jonathan has a typical response when confronted.  He restates the issue, gives excuses that don’t directly address the question, often will say we should remove the questionable data and reanalyze, and then finally will agree to retract if the results don’t hold. This was the pattern until lawyers got involved that is.  

[…] When I was early in my career I was also just enamored with him and trusted him.  So when he comes dangling a dataset that would turn into a high level paper, you gladly take it and write it up.  Later it seemed like what would happen is that he’s send off just chunks of datasets to different people so they don’t see the whole thing.  Or he would talk to say a postdoc about an idea, the six months later the data would appear in that postdocs inbox….  so they don’t ever see the stuff being collected, but are thrilled to have the data and to get a famous person to work with.” 

Pruitt raw data file “Spider_StateBasedPersonality”, sheet “Anelosimus.2” Colour coded are repetitive number sequences

Much of #PruittData problems came out because Pruitt used to share his research data openly. Until he got caught that is:

Dryad is the data repository for open source stuff. Many journals have started requiring us to upload our data. Jonathan would do that sometimes, but not others. Or he would upload only a portion of the data. This started when one of his own grad students acted as a whistleblower and told another lab about some issues. That lab looked into a bunch of his data on Dryad and found all these issues.

But now, the lawyer’s letter. Which even Pruitt himself admits is not suitable to scare anyone. He namely wrote to me via Twitter direct message, obviously with reference to the Science article:

I realize the report made it sound like I legally carpet bombed people; that’s not the case. I know folks have interpreted the action as a flimsy fear-induced gag order. That’s not the case either. I’m happy for folks to engage in public discourse about my data integrity


PRIVATE & CONFIDENTIAL
WITHOUT PREJUDICE

[….]

Please be advised that our firm has been retained by Jonathan Pruitt with respect to the investigation (if any) being conducted by […] into the paper “[…]” by […] in 20[…]. I understand you have been contacted by Dr […] one of the coauthors, seeking a retraction.

Over the last number of weeks, there has been an online campaign directed at Dr. Pruitt and his work which has come to be known as “#pruittdata” and “#pruittgate”. As part of #pruittgate, a number of academics have organized an extra-institutional online forum for posting, reviewing, and criticizing the data integrity of various of Dr. Pruitt’s research studies and papers. It appears that Dr. […]’s concerns arise from that project.

Dr. Pruitt disagrees that there are any grounds for this paper’s retraction. Any decision about retraction must be conducted fairly and in keeping with professional norms as articulated by the Committee on Publication Ethics (“COPE”) Guidelines on Retractions. It does not appear that this paper meets any of the criteria for retraction as set out in the COPE Guidelines. In particular, the COPE Guidelines do not support retraction if “the main findings of the work are still reliable” or if “an editor has inconclusive evidence to support retraction, or is awaiting additional information such as from an institutional investigation.”

In addition, I am concerned that the procedural fairness of any investigation has been hampered by the manner in which Dr. Pruitt’s work is currently being criticized online. The social media campaign and online forums which spurred this complaint break with the normative practices of the field and very likely prevent a fair assessment of Dr. Pruitt’s work.

The controversy over this particular paper highlights the problems with using an unmediated online forum to attack the data integrity of a body of work. In this case, it appears that the wrong electronic file was uploaded to the forum. The final published paper does not rely on the impugned data.

If […] there is sufficient grounds to retract this paper, it must first notify Dr. Pruitt and provide him with a chance to respond. A premature retraction without Dr. Pruitt’s agreement is likely to further harm his reputation and negatively impact his career in an irreversible manner.

Please confirm that […] does not intend to take further steps at this time
with respect to the above noted paper. Otherwise, if the journal intends to take any further steps, please provide me with correspondence (1) detailing the journal’s concerns, including what evidence it has showing that the paper’s findings are no longer reliable, and (2) outlining the process the journal intends to follow prior to retraction, including when and how Dr. Pruitt will be permitted to respond to any allegations against him.
I am happy to discuss this matter further if you have any questions or concerns. If you would like to speak to me, please arrange a time through my assistant Lori Leblond (lori@millardco.ca, 1-416-920-3123).
Yours truly,

Marcus McCann

Update 19.05.2020. Seems Pruitt’s lawyer was successful. The Pruitt et al Proc Roy Soc B: Biol Sci 2016 paper, previously retracted on 26 February 2020, was stealthily de-retracted by the publisher The Royal Society in March. Instead, an Expression of Concern was published on 29. April 2020, announcing “An investigation into these aspects is under way“. This is the deleted retraction:


Donate!

If you are interested to support my work, you can leave here a small tip of $5. Or several of small tips, just increase the amount as you like (2x=€10; 5x=€25). Your generous patronage of my journalism will be most appreciated!

€5.00

15 comments on “Spiderman’s lawyer is having you for dinner tonight

  1. Somebody should inform spiderboy’s lawyers:

    Like

  2. Pingback: Here’s the letter Jonathan Pruitt’s lawyers have been sending to journals and his former collaborators #pruittgate #pruittdata | Dynamic Ecology

  3. Though Laskowski is completely innocent, it is not unreasonable to speculate that she got a faculty job from his questionable data, from which she got three published papers from, and she may have pivoted off that for more work. This gave her an advantage over other candidates that did not have her great publication record. I dont, however, seeing her rescind her faculty position! Why didn’t she collect her own data? Most of my ideas are wrong from the data that I collect myself, and I kind of wish the same thing happened to her to equal the playing field. But then again, maybe if this had happened, a full fledged cheater who fudged data could have then gotten the position.

    Like

  4. Pruitt’s co-author Nicholas DiRienzo now commented on PubPeer
    Pruitt et al Behavioral Ecology and Sociobiology (2011)
    https://pubpeer.com/publications/2BE2F156C5AC7C0C33939A612FDD9C#1
    https://drive.google.com/uc?id=1grJs92VKblsNro0rKGRKNRETuRakZUz7
    Pruitt explained it as

    “Lingering formula are present in some versions of data files when there contained missing values. Excel-based stats and other older stats programs don’t always contend with them well. It’s possible I uploaded a version with one of those present. […] If they change the results in an important way that changes the interpretation, then ax the paper/s. If not, then consider an addendum. It’s important not to throw the baby out with the bath water.”

    DiRienzo concludes

    “Myself and the other coauthors (excluding Dr. Pruitt) deem the data associated with this paper as being untrustworthy. There is no reason that such a formula should ever exist in a dataset that is supposed to contain individual behavioral measures, and I have yet to receive a satisfactory explanation as to why it was in there. Additionally, the discrepancies between the values contained in the Small and Large datasets cannot be explained by any simple excel or data input error. We have requested a retraction from the journal and it is currently being reviewed by the editors.”

    Like

  5. Looks like Pruitt’s lawyer letter may have had a chilling effect after all:

    https://pubpeer.com/publications/F62202DB4DB0804CD2E65C25764549#8

    It seems reasonable to draw a connection between the lawyer letter and the apparent posting and removal of a retraction notice.

    Like

  6. Pingback: Interview with JBC research integrity manager Kaoru Sakabe – For Better Science

  7. Pingback: Mr ACE2 Josef Penninger, Greatest Scientist of Our Time – For Better Science

  8. Klaas van Dijk

    Copy/pasted from https://www.sciencedirect.com/science/article/pii/S0003347213002492 :

    “RETRACTED: Linking levels of personality: personalities of the ‘average’ and ‘most extreme’ group members predict colony-level personality, Jonathan N.Pruitt, Lena Grinsted, Virginia Settepani

    This article has been retracted at the request of the authors LG and VS. It has come to the authors’ attention that the data on spider boldness reported in this paper appear to contain irregularities in the form of an excess of duplicated values. These duplicated values even span across time points. For example, the boldness value 63.21 sec occurs in both boldness 1, 2 and 3 which were assayed at different times. Furthermore, extensive overlap has been found between this data and that from a different article published the same year: 74.3% of the boldness values are identical to those found in the other publication even though different spiders were assayed. On this basis, the authors LG and VS have lost faith in the reliability of the data and they wish to retract the paper. Author JNP does not agree to the retraction.”

    Like

  9. Klaas van Dijk

    Copy/pasted from https://royalsocietypublishing.org/doi/10.1098/rspb.2020.0255 :

    “Retraction: The Achilles’ heel hypothesis: misinformed keystone individuals impair collective learning and reduce group success

    N. Pinter-Wollman , C. M. Wright , C. N. Keiser , A. DeMarco and M. M. Grobis

    Published:29 July 2020

    The authors of the paper by Pruitt J.N., Wright C.M., Keiser C.N., DeMarco A., Grobis M.M., & Pinter-Wollman N. `The Achilles’ heel hypothesis: misinformed keystone individuals impair collective learning and reduce group success’ published in 2016 in Proceedings of the Royal Society B283: 20152888 were recently made aware of problems in the raw data collected by Pruitt J.N. In particular, 86% of the values of the rewarding stimulus in time 9 are separated by an integer (11) from the values in time 11 of the rewarding stimulus. Similarly, 82% of the values in time 10 of the rewarding stimulus are separated by an integer (3) from values in time 12 of the rewarding stimulus. In addition, 21% of the values in time 2 of the rewarding treatment are separated by a fixed integer (8) from values in time 1 of the dangerous treatment.

    Furthermore, in the data on individual learning, 80% of the values from time step 5 in the rewarding trial are separated by exactly 10 from the third time point in the unrewarding trials. Similarly, 80% of the values from time step 6 in the rewarding trial are separated by exactly 11 from the second time point in the unrewarding trials. These irregularities are focused on the keystone and generic individuals and no such irregularities are found in the control individuals.

    Reanalysis, with these irregularities removed, no longer provides clear support for some of the main conclusions of the study. The authors therefore wish to retract the paper.”

    Note that Jonathan Pruitt is not listed as author of this retraction note and note that this information is not listed in this retraction note. This retraction note also does not provide details about the whereabouts of the raw data which were removed because of “irregularities” / “problems”.

    Like

  10. Klaas van Dijk

    And another Retraction (published in July 2020). Copy/pasted from https://www.sciencedirect.com/science/article/pii/S0003347220301342 :

    “Retraction notice to “Individual differences in personality and behavioural plasticity facilitate division of labour in social spider colonies” [Animal Behaviour 97 (2014) 177–183]

    This article has been retracted at the request of the Editors and authors. After scrutinizing the raw data associated with this paper, the first and second authors discovered irregularities in the form of duplicated data sequences. In particular, identical behavioural profiles were repeated across individual spiders belonging to colonies that were purported to be independent. As these anomalies cannot now be explained, these authors consider the results to be unreliable. They have, therefore, retracted the paper. The data were collected by the third author, who agrees to the retraction.”

    Once again “irregularities” and “anomalies” in data which were collected by Jonathan Pruitt and which could not be explained (by Jonathan Pruitt?).

    Like

  11. Klaas van Dijk

    And an Expression of Concern of another paper in Animal Behaviour. Copy/pasted from https://www.sciencedirect.com/science/article/abs/pii/S0003347220301706 (published in August 2020):

    “Several concerns have been raised about the study’s integrity and data validity. This expression of concern is to inform readers about the potential issues related to this article. An investigation has revealed that the boldness measures, although not used in this study, contain some anomalous patterns of decimals in recorded latencies. These measures were supposed to be taken with a stopwatch, and thus one would expect the decimals that were recorded to the hundredth to be effectively pseudo-random. Yet, the centi decimals present are commonly repeated. Simulations show that these patterns of repeats should virtually never occur naturally, which raises questions regarding their validity and accuracy. Although this suspect measure is not used in the paper, it was collected as part of the aggression trials, of which other measures were used. Dr. Pruitt has not provided a sufficient explanation for this issue. Given this, Drs. DiRienzo and Hedrick would like to make the readers aware of these unexplained data anomalies. For a full description of the history of the datasets and the investigation done by Dr. DiRienzo, please see this post on PubPeer:https://pubpeer.com/publications/00E431B139DAF33455AC1FC2418F0F

    First author Nicholas DiRienzo wrote on 23 March 2020 at this Pubpeer thread:

    “I agree that the small dataset that is mainly integers makes it hard to detect anomalies (which is why I focused on the boldness data). I do find the three mass/boldness increases curious. I hadn’t found those, but I was trying to avoid sorting the data every to reduce the chances of finding false positives.

    It’s hard for me to give this portion of the dataset an endorsement beyond “I think they’re trustworthy.” Another way I could put it is “I couldn’t find any giant red flags, but I’m skeptical.” I made these posts as science demands trustworthy data, and thus even just even having to ‘think’ if they’re trustworthy should give readers pause.”

    First author Nicholas DiRienzo wrote on 5 August 2020 at https://twitter.com/Niku_DiRienzo/status/1290997804706328576 :

    “1 of 5 resolved. Not a retraction, but did get a Expression of Concern published highlighting the data anomalies. Please read the description below and check out the PubPeer post for more details.”

    “I think Animal Behaviour has been incredibly bold. There’s been a lot happening behind the scenes and they’ve powered through everything to do what’s best in each situation. Now, other journals… well, some have had their own best interests in mind.”

    “The other issue is that some are treating each paper as an isolated incident as they’re afraid of getting sued even though they should consider the whole of the situation.”

    “Some are also only concerned about the influence of the anomalies on the results. If they don’t hold with the anomalies removed then they retract, otherwise they just allow (force) a correction. This is why we’re seeing corrected papers with half the data removed.”

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: