In June 2023, Maarten van Kampen published his analysis of the mysteriously duplicated numbers inside the raw data in the papers by two neuroscience professors at Stanford University: Thomas Südhof, laureate of the 2013 Nobel Prize in Physiology or Medicine, and his wife Lu Chen.

It turned out that the raw data published as supplemental material was actually not raw data at all: it was created post-hoc, when the manuscript was submitted for publication. Hence, when the authors were caught with forged replacement data to fix the duplications, this was not relevant for the traditionally unaffected conclusions, and Südhof declared:

“Yup we are being hounded.”

Still, Chen’s paper recently received a mega-correction for what she insisted were “careless but honest mistakes“, and Südhof was about to correct his lab’s paper, but the mounting evidence of fraud apparently got him to reconsider. A retraction is likely.

For backstory, read this earlier article by Maarten.

And now, his second attempt at hounding.

---

A second attempt at bookkeeping

By Maarten van Kampen

In an earlier ‘Bookkeeping’ post I dissected two publications from the labs of Lu Chen and Thomas C. Südhof. Both papers were flagged on PubPeer for potential duplications in their underlying data, showing surprisingly large numbers of values repeating in- or between conditions.

The authors’ initial reaction to these concerns was rather surprising. The first author of the Südhof paper responded by mostly acknowledging the issues and providing fabricated replacement data. Lu Chen acknowledged the first concern, but rejected the second by providing impossible original data which I now deem likely to have been fabricated as well.

In the four months since the initial For Better Science post both groups have prepared corrections. The correction for the Lu Chen paper has been published, the one for the Südhof paper is announced. Here I want to shortly discuss the former and give my expectations of the latter.

First, a few lines of introduction. The authors of both papers study firing patterns of neural cells. They measure the current spikes emitted by a neuron versus time as shown in the traces in panel (a) below. Each trace is quantified by determining the firing frequency (b) and spike amplitude (c). The frequencies and amplitudes thus come in pairs and each ‘open circle’ data point in the frequency plot of panel (b) has a counterpart in the amplitude plot in panel (c).

Retinoic acid paper / Lu Chen

Shruti Thapliyal, Kristin L Arendt, Anthony G Lau, Lu Chen Retinoic acid-gated BDNF synthesis in neuronal dendrites drives presynaptic homeostatic plasticity eLife (2022) doi: 10.7554/elife.79863

In the Correction from 2 August 2023, the authors appear to have completely re-evaluated their data. All suspect repeats identified on PubPeer are replaced, as are some additional repeats that were not flagged. The supplementary data files and the figures in the paper have been updated. The correction comes with a painstakingly complete description of the convoluted data processing workflow and an explanation of how this led to the duplications. This appears to be a well-done correction.

Having said that, I still have very strong reservations about the ‘original data’ that the authors provided on PubPeer. That data was used to disprove the now-accepted bookkeeping errors:

The above author-supplied table shows that the eight apparent duplications between the yellow columns are not duplications at all. For one, the corresponding firing frequencies are different (‘Freq. of Cell’). This shows that the Fig. 3e and 3f numbers cannot come from the same measurement as they do not form amplitude-frequency pairs. Secondly, the authors found that actually also the amplitudes are different. This magic happened when they looked at the ‘extended decimals’ that were only available in their original data.

The above duplications are now accepted as a copy-paste errors and corrected. One could charitably believe that the mismatch in the ‘Freq. of Cell’ values in the author-provided original data is due to further disorganization that remained unrecognized even after the authors were warned. But neither bookkeeping nor copy-paste errors generate extra decimals. And incidentally every first unpublished decimal makes the numbers different, be it for numbers that were published with 3, 4, or 5 decimals behind the point. For example, 11.163 was actually 11.1627 or 11.1631. And 10.42005 was either 10.420051 or 10.420049.

Lacking any other viable explanation I judge it most likely that the author-provided original data was a fabrication to alleviate concerns of critical peers.

Neurexin-2 paper / Südhof

Pei-Yi Lin, Lulu Y. Chen, Peng Zhou, Sung-Jin Lee , Justin H. Trotter, Thomas C. Südhof Neurexin-2 restricts synapse numbers and restrains the presynaptic release probability by an alternative splicing-dependent mechanism Proceedings of the National Academy of Sciences (2023) doi: 10.1073/pnas.2300363120 (“Contributed” by Thomas C. Südhof as NAS member)

The authors of the Neurexin-2 paper received the first concerns on 25 March 2023, only one day after the work was published by PNAS. The first author responded to the PubPeer concerns in less than two hours, serving her critical peers fabricated replacement data. Two examples are given below:

For the ΔCre column (left panel) the first author ‘removed’ repeating sequences of frequency values by simple row-wise addition or subtraction of 0.2 and 0.3. The sum of these changes was zero, nicely preserving the mean. In another fix the author combined ±1 and ±2 ‘corrections’ (see red rectangle, bottom-right) with a complete re-ordering of the data (SS4+SS5-, right panel).

The issue with the repeat sequences was pointed out in March 2023, the fabrication of replacement data in June this year. Then, on 8 August 2023, the senior author announced an upcoming correction (highlights mine):

“These mistakes were due to copy-paste errors that occurred when the data files were compiled from individual data files during assembly of the summary file after the data had been analyzed, the figures had been made, and the paper had been written. Since the figures and conclusions of the paper were generated before compilation of the summary file, the copy-paste mistakes in the summary file did not enter into the paper and the mistakes we made in the summary file have no effect on the actual paper. […]

We have now submitted an erratum to PNAS to correct the record. […] We hope that the availability of original data will clarify the issues raised by the criticisms voiced above and again apologize for the errors in the summary data file. Sincerely, Tom Sudhof”

The above is an exceptionally fortunate outcome: the errors that were identified on PubPeer are the result of copy-paste errors that are solely present in the summary Excel file. And hence the conclusions and figures of the paper can remain as-is: only a new supplementary data file needs to be uploaded and the whole affair can be left behind. The fabricated replacement data is not addressed, but this is maybe of no consequence as it was only published on PubPeer.

The correction is not yet published, which makes it somewhat difficult to criticize it. What Südhof tells us in his announcement is that the data and figures in the paper are correct. But it is not that simple, and I will explain why his planned correction cannot stand.

The following issues are apparent, either related to bookkeeping errors or simply fabrication:

1. Impossible repeats in the mEPSC frequency data of Fig. 6(b)
2. Fig. 2 and Fig. 6 mEPSC data are inconsistent
3. Fig. 4 and Fig. 6 NDMAR-EPSC data are inconsistent

Impossible repeats in Fig. 6(b)

This concern should sound familiar. It was raised in March 2023 and ‘fixed’ by the fabrications that were just discussed above. But, in a cynical turn of events, these fabrications are now deemed unnecessary. The as-published Fig. 6(b) frequency data corresponds perfectly to its supplementary data table that showed these many suspicious repeats:

Initially I assumed this meant that Fig. 6(b) is affected by copy-paste errors. The senior author Südhof however assures us that in this case both the figure and supplementary data are correct:

“In particular, the data criticized here are from mEPSC measurements that have discontinuous values. Thus, for each conditions these data represent repeats of a limited number of values – these are NOT copy-paste errors or duplications but this is inherent in the experimental setup“

But the above is simply so extremely unlikely that it can be safely deemed impossible. The figure below shows the table behind Fig. 6(b), with repeat sequences of 5, 6, and 7 consecutive values highlighted in respectively blue, yellow, and red:

The firing frequencies are derived by dividing the number of observed spikes by the duration of the measurement. This makes the chance of measuring the exact same frequency much higher than one would expect based on the number of decimals in the above frequency values. And this is exactly the point the senior author makes in his PubPeer comment.

Still, in this PubPeer post I come to at most a 1% probability of measuring the same frequency twice in succession. With that value one can estimate the probability of finding a single repeated sequence of a given length, accounting for us searching throughout the whole table:

The longest, 7-value repeat thus has a one in seven billion probability of occurring. Meaning, what we observed in this paper can utterly unlikely have happened by chance. The six value repeat is some 100x more likely to happen, but at one in seventy million it is still a statistically very significant ‘find’.

One can only conclude that there is something seriously wrong with the data in this table, and hence with paper. And below we will get some clues as to what really happened…

mEPC data with bookkeeping errors

The paper shows two figures with mEPC data. One of these is Fig. 6(b) and (c) discussed above. And in the insets of Fig. 2(f) one can additionally find mEPC frequency and amplitude data for the ΔCre and Cre groups:

The number “11/3” in the Fig. 2(f) ΔCre column means that the authors took 11 measurements on 3 separate tissue slices. In Fig. 6(b) we find 20/4, so one more tissue slice was used to obtain nearly the double amount of measurements. It is principally possible that the 11 measurements of Fig. 2(f) are part of the data used in Fig. (6). So let’s check whether we can find shared amplitude and frequency values between these datasets…

In the figure below I graphically compare the mEPC amplitude data between Fig. 2(f) and Fig. 6(c). Comparing the vertically scaled version of Fig. 2(f) in the middle to Fig. 6(c) on the right one can see little correspondence. For example, about half of the amplitudes in Fig. 2(f) lie below 10 pA while (nearly) all amplitudes in Fig. 6(c) lie close to 15 pA, i.e. at least 50% higher. Also a more detailed comparison below shows no overlap, not even when scaling the data ¹.

The amplitude data for both figures is thus derived from completely independent measurements. And given the ‘pairing’ of amplitude and frequency values the same should thus be true for the frequencies. So let’s compare the frequency data:

The frequencies reported in the supplementary data for Fig. 2(f) match perfectly with the data in the figure, compare the blue dots with the open circles in the left panel above. And we already found that the same is true for the data in Fig. 6(b), reproduced in the right panel. As a result, we can just look at the numerical data:

And that data shows something very surprising. All eleven ΔCre frequency values of Fig. 2(f) appear in the exact same order at the bottom of the ΔCre column of Fig. 6(b). The Cre column shows a more complicated pattern, with all but two values duplicated at the top of their column in Fig. 6(b).

The observations above already show that, despite the author’s assurances, the as-published paper was struck by at least a few more copy-paste ‘errors’: if the data in Fig. 2(f) would have been a subset of that in Fig. 6, then the frequency-amplitude pairs would have been duplicated. Instead, we only find duplicated frequencies. And there is simply no valid reason for using the frequency values of 21 earlier measurements, but leaving out their ill-fitting amplitude values. And these amplitude values are purportedly substituted by those from new measurements whose frequency values are for unknown reasons left out…

The situation is actually quite a bit worse, with even some hints of a motive. As was already noted, with the addition of only one extra tissue slice the amount of measurements in the ΔCre nearly doubled between Fig. 2(f) and 6(c). Looking in more detail at that column one finds a sequence of four values duplicated (blue numbers in the table below). Together with the single duplication of a 2.2 Hz value in red this only leaves the first four values unrelated to those in Fig. 2(f). One could surmise that of the nine extra ΔCre frequency values in Fig. 6(b) only four were actually measured, with the other five being simply duplications of Fig. 2(f) data.

And there is still more. The main premise of this paper is that by knocking out the Neurexin gene the mEPSC firing frequency changes. This knock-out can be ‘healed’ by adding back both the SS4 and SS5 gene. Our ΔCre column above represents the control group and the SS4+SS5+ column at the right the ‘healed’ mice. The more the SS4+SS5+ frequencies look like the control, the more convincing the story. And when looking in detail, these frequencies indeed look very much the same.

The repeated sequence highlighted by the red box is the ‘one in 70 million’ 6-value repeat that was already highlighted in the ‘Impossible repeats’ section. But there is also a 4-value sequence within that repeat that is itself repeated in the SS4+SS5+ column (green highlight). As a result, 10 of the 25 frequency values in the SS4+SS5+ column are shared with both the Fig. 6(b) and the Fig. 2(f) controls. This is extremely implausible, but yes, it certainly helps making the SS4+SS5+ intervention look successful.

NDMAR-EPSC data with bookkeeping errors

More of the same to follow; when you are bored, skip to the conclusions if you like!

Here we will look at ΔCre NDMAR-EPSC amplitude and paired-pulse ratio data plotted in Fig. 4(e/f) and Fig. 6(e/f):

Also here one could expect the Fig. 4 data to be a sub-set of the Fig. 6 data. And after checking whether the supplementary data tables correspond to the plotted values (they do ²) one can have a look at these tables. Let’s keep to the ΔCre values:

For the NDMAR-EPSC amplitude data we find that the author has re-used nine measurements between the ΔCre columns of Fig. 4 and Fig. 6 (green shades & red). The values are re-ordered but still occupy a continuous row 2 to 10 stretch in Fig. 6(e). The SS4+SS5+ group is mostly spared from coincidences.

When looking at the paired-pulse ratios (PPR) we see something strikingly different. There are zero matching numbers between the ΔCre columns of Fig. 4 and Fig. 6. Instead, we find a sequence of 4 values repeated between Fig. 4 ΔCre and Fig. 6 SS4+SS5+ (yellow highlight). On top of that, two other values can be found duplicated. Noteworthy: all six values ‘migrated’ from the ΔCre healthy control group of Fig. 4 to the ‘successfully treated’ SS4+SS5+ group of Fig. 6.

The above cannot have happened by chance. The Fig. 4 data is clearly not a subset of the Fig. 6 data as the amplitude and paired-pulse ratio pairing is completely broken. At the same time the datasets are not independent: a large number of amplitude values is shared between the ΔCre columns, while paired-pulse ratios are shared between the ΔCre and SS4+SS5+ columns. And having 6 of the 22 values in the SS4+SS5+ column duplicated from control measurements will again help inflate the effect of the claimed intervention. I have a hard time to accept this as honest mistakes.

Of note: this mix-up between control and SS4+SS5+ group was already pointed out in April by Scutellinia korfiana. That user even send a kind reminder to the authors after the concern seemed to have been ignored.

Conclusions

The authors of both the Retinoic acid and the Neurexin paper seem to have tried to alleviate the concerns raised on PubPeer by providing fabricated data. This is abundantly clear for the Neurexin authors; for the other paper it may simply be my lack of imagination when it comes to bad bookkeeping.

The authors of both papers have submitted corrections to the respective journals. The Retinoic acid correction in eLife appears exemplary in its detail and corrects more issues than were pointed out on PubPeer. The Neurexin correction should never be accepted by PNAS.

I raised the above concerns both on PubPeer and with the publisher and believe them to be convincing. And now even the senior author of the paper is starting to realize that something is amiss. In August Südhof’s response to the concerns raised on PubPeer was a repeated:

“Please analyze the raw data. We cannot base our conclusions on statistical probabilities, we have to rely on actual data.”

More recently, on 27 September 2023 and a bit over a month after the last concern was raised, the senior author posted the following (highlights mine):

 “We agree that in addition to the multiple copy-paste errors in the original source data file, the uneven selection of shared vs. non-shared data for the controls in some Figures did not meet the standards of scientific rigor that we aim for, although this is not in itself against the rules if shared data used in different experiments as long as the experiments were performed at the same time in the same batch with the same controls. Naturally if only a subset of shared control data are selected for analyses, they have to be randomly and evenly selected.  We concur, furthermore, that the ‘replacement source data’ provided by Dr. Pei-Yi Lin, the lead author of the paper, are likely erroneous.”

If one was cynical, once could consider this to be maneuvering towards an impending retraction, with as an opaque as possible retraction notice ³. Retracting a paper because the underlying data is fraudulent hurts, especially when you realized the fraud only after some six months of external pressure whilst insisting on the “wait and look at the raw data” approach. It is obviously much nicer to write that one retracts a paper because “the uneven selection of controls in some figures did not meet the high standards of scientific rigor that we aim for“. Meaning, a good paper is retracted for not being excellent.

In the last bit the senior author acknowledges there was something wrong with the replacement source data provided on PubPeer. I hope this makes the authors also more seriously question the data presented in the paper itself.

In all, I think this is an as-happy end as it gets when raising concerns.

---

1 . I am not a neuro-biologist, so maybe I am saying something stupid here. The paper actually shows three sets of mEPSC amplitude data: Fig. 2(f) and 6(c) discussed above and Fig. S1(f). The latter is a replication by a second researcher, presumably to prevent confirmation bias or maybe even fraud. In the figure below I compare the reported mEPSC amplitudes. Each figure shows a largely different average amplitude of respectively ~10, ~15, and ~20 pA. These differences are very significant: the standard error of the mean is 1-2 pA. How come these experiments are internally so consistent, yet differ so much between each other? It obviously also makes it rather impossible to aggregate the amplitude data.

2. Long comparison of the supplementary data with the plots in the published paper for the ΔCre and Cre groups in Figs. 4(e/f) and 6(e/f). The values in the supplementary data for these groups correspond to those in the published figures as shown by the perfect match between the black circles from the published figures and the overlaid red circles representing the values from the supplementary data.

3. I am for example thinking of Nature’s retraction of Ranga Dias first “Room-temperature superconductivity” paper. Instead of calling out the paper for fabrication, the retraction notice states: “the background subtractions applied … used a non-standard, user-defined procedure”. This likely helped the same authors to publish their second room-temperature superconductivity Nature paper earlier this year. And, ironically, it seems the same “background subtraction” reason can be repeated for its upcoming retraction.

---

PS by LS

I wrote to Südhof (with Chen in cc), asking if he indeed announced to retract the Neurexin paper. Nobody replied to me, but right after my email, Südhof posted this on PubPeer:

“In response to my last post, I was asked by Dr. Leonid Schneider, who together with Dr. van Kampen (a commentator on this paper on PubPeer) wrote a critical ad hominem blog discussing my family, whether my latest post means I intend to retract this paper. This is not the case, although it is possible that the lab as a whole will recommend a retraction after a detailed review of the raw data. PubPeer is not a peer-reviewed official publication of my lab. If a lab member posts something on PubPeer in her or his own capacity, that is within her or his first-amendment rights but such postings are not lab publications. All I stated in my comment was that the replacement data provided for this paper that were discussed extensively were not from my lab. At this point we are reanalyzing the raw data for the files that contain errors and will assess the validity of the conclusions. We will then decide whether the scientific rigor of the original analysis suffices to support the conclusions of the paper.”

I have several issues with this reply, here are some of them:

Maybe as an immigrant to US, Dr Südhof is not yet fully aware of what the First Amendment really means. It grants a right to unrestricted free speech, but does not free you from responsibility for what you say or write, especially in public, especially in professional capacity. If one posts fraud like his lab members do, one can be called out even if this platform was “not a peer-reviewed official publication“. Be it on PubPeer, in a preprint, or in an NIH research grant application.

Finally, I believe it is not up to Südhof and his lab of irresponsible equals alone to decide whether or not their paper should be retracted or not. But for the scientific community, which is why Maarten and myself published this second “ad hominem” post.

---