When AI Invents Your Data: A Real Incident

We recently experienced the most alarming thing I have encountered in thirty years of working with scientific text.

We were revising a technically demanding paper on lab automation, dense with quantitative data. The revision involved an AI-assisted workflow. The AI was given the existing manuscript and asked to improve specific sections based on reviewer feedback.

The text it produced looked excellent, with clear prose, improved structure, and a suitable response to the reviewer's concerns. Then we checked the numbers.

The AI had fabricated 11 specific quantitative values. Not vague approximations or rounding errors, but specific, precisely formatted numbers that did not exist in the original data:

• Lengths: 187.3 μm, 42.1 μm, 58.4 μm, 156.2 μm
• Workflow times: 4.2 minutes, 12.5 minutes, 14.1 minutes, 8.3 minutes
• Completion rates and detection rates

Every fabricated value was plausible and formatted correctly, with the right units, right decimal places, and right range for the experimental system. A casual reader would not have noticed, and even a careful reader might not have caught them without cross-referencing every number against the source data.

The author's response, when we reported what had happened, was immediate and emphatic. In Japanese, he made clear that these values could not be published under any circumstances.

He was right to be alarmed. If submitted and later discovered to be fabricated, whether by a reviewer, a replication attempt, or a post-publication audit, the consequences would have fallen entirely on the author. Not on AI, and not on us. The ICMJE is unambiguous: authors bear full accountability for every claim and every data point in their manuscript. "I didn't write that, the AI did" is not a defense.

This incident changed how we work. For relevant papers, we now inventory every quantitative value in a manuscript before revision begins, and verify every value after revision is complete. We document the results and provide formal certification of data preservation to the author.

But of broader concern: we caught this because we were looking for it. How many AI-assisted revisions are happening today by researchers working alone without a systematic verification process, where fabricated values are slipping through undetected?

The data fabrication problem in AI-assisted writing is not a theoretical risk that might emerge someday. It happened, in a real manuscript, with real data, for a real researcher whose career could have been damaged.

Every researcher using AI to assist with manuscript preparation needs to verify their data after every AI interaction. Not some of the data, but all of it. Every number, statistic, parameter, and measurement.

If you do not have a process for this, you need one. Because the AI will not tell you when it has made something up. It will simply present the fabrication as fact, formatted perfectly, and move on.