Why AI Writing Detectors Break at New Model Generations, and What arXiv's Caps and Hexagon Mean for Research Publishing
An October 2026 study finds a detector frozen at GPT-4 catches at most 0.6% of GPT-5-era rewrites, while arXiv capped submissions and mathematicians launched Hexagon for AI-generated proofs. What journals, labs and editors should change.
AI-writing detectors used to screen scientific manuscripts can stop working almost completely when a new model generation arrives. A study posted to arXiv in October 2026 found that a detector trained on GPT-4 rewrites caught at most 0.6% of rewrites produced by any GPT-5-generation model. In the same month, arXiv introduced per-author submission caps after a surge in volume, and a group of mathematicians launched Hexagon, a repository that accepts AI-produced proofs no human claims to understand. This article explains what the study measured, why the failures cluster at generational boundaries, and what publishers and labs can do about it.
What the study measured
The paper, “Large Language Model Turnover Undermines Screening for Artificial Intelligence-Assisted Scientific Writing”, took abstracts from the Proceedings of the National Academy of Sciences published before ChatGPT and had them rewritten by 23 model versions from three vendors (OpenAI’s GPT family, Alibaba’s Qwen and Meta’s Llama and Muse), released between June 2023 and August 2026. It then trained detectors under different maintenance scenarios and tested whether they could flag rewritten abstracts while falsely flagging only about 1% of genuine human abstracts.
The focus on rewritten rather than fully generated text matters. Rewriting is the common real-world case, where an author asks a model to polish an existing draft, and it leaves fewer obvious traces.
Key findings
Frozen detectors fail across generations, but unevenly. A detector trained only on GPT-4 caught at most 0.6% of rewrites from GPT-5 onward, and only 0.2% of GPT-5.4 rewrites. By contrast, a detector frozen at Qwen2.5 still caught 86% of Qwen3.8 rewrites. Ageing alone does not explain the failures; specific generational jumps do.
Regular retraining mostly works, with two exceptions. When the detector was retrained on all earlier versions before each new release, 18 of the 20 versions with earlier data stayed above 85% detection (median 95.4%; lowest 85.5% for GPT-5.1). Detection collapsed twice:
| New version | Detection when trained on earlier versions only |
|---|---|
| GPT-5 (first of the GPT-5 generation) | 26.6% |
| Muse-Glimmer (first open-weight model in Meta’s Muse family) | 3.8% |
| Qwen3 (new architecture) | 94.7% |
| Llama 4 (mixture-of-experts architecture) | 99.2% |
Architecture changes such as Llama 4’s move to mixture-of-experts did not by themselves break detection, while some new generations did.
Vendor names are a poor guide. A detector trained on GPT-4o (November 2024) caught 99.5% of Llama 3.3 rewrites, a contemporary model from a different vendor, while the same-vendor GPT-4 detector failed on GPT-5.4.
It is not a classifier quality problem. The same collapse appeared with a simple word-frequency detector that uses no neural network, at operating points of 5% and 0.5% false positives, and with two alternative rewrite prompts. That suggests the vocabulary of new generations really shifts, and a better classifier alone will not fix it.
One combined detector or one per version?
The authors compared two screening designs, each calibrated to flag about 1% of human abstracts:
- A union of per-version detectors flagged 1.1% ± 0.4% of held-out human abstracts and had a median detection rate of 99.3% across the 23 versions.
- A single pooled detector flagged 0.8% ± 0.3% of human abstracts with a median of 97.2%.
The gap was largest on Muse-Glimmer (83.6% union versus 66.2% pooled). The union approach catches more but costs more to maintain and flags slightly more innocent authors. At around 1%, both still flag roughly one in a hundred human-written abstracts, which is why any detection result should lead to human review rather than automatic rejection.
arXiv’s submission caps
As TechTimes reported, arXiv announced on October 1, 2026 that authors are limited to two submissions per calendar month and three active submissions at a time, describing a “watershed moment” in AI-enabled paper volumes. It received 40,363 submissions in September 2026, more than four times the volume of the same month a decade earlier, and its volunteer moderators were overwhelmed. Submitting entirely LLM-generated content was already prohibited.
A cap controls volume; it does not detect anything. A rewritten abstract can arrive well within the limit, and the study above shows why a screen trained on older models may not notice it. For labs, the practical effect is that splitting one project into many short preprints is no longer an option.
Hexagon: a home for proofs nobody claims
The same TechTimes article describes Hexagon, a repository for research-level mathematics and theoretical computer science announced on October 6, 2026 by Northwestern mathematician Benjamin Antieau and cross-posted by Fields Medalist Terence Tao. It explicitly accepts submissions where no human claims to understand the result. Antieau’s motivating example was a real result about Hodge numbers produced by a language model that he was unwilling to sign as his own work; arXiv, which requires human intellectual contribution, told the group it would not host such results.
Hexagon tries to keep accountability through disclosure. A human submitter with a verified ORCID account files each work, and every submission carries two tags: how much of the text is AI-generated, and whether human understanding is claimed for all, some or none of it. It is not intended as a dumping ground for papers arXiv rejected; its stated purpose is results that may be real and checkable but have no human author.
What journals, labs and editors should do
- Record the detector’s training cutoff next to every score. A probability with no information about which models the detector has seen can be meaningless after a generation change.
- Retrain when a major generation ships. The study shows most in-generation releases stay detectable with retraining, but there is no grace period at a boundary such as GPT-5.
- Keep a simple baseline, such as a word-frequency detector. If it drops at the same time as your main classifier, the cause is a vocabulary shift, not a bug.
- Publish the false-positive rate in author-facing policy and state whether the screen is pooled or a union.
- Use detection to trigger human review, never automatic rejection.
- Ask vendors for results on the newest generations, not only on GPT-4o-era rewrites, and for each detector’s training cutoff.
For more on how model behaviour complicates research evaluation, see our coverage of models omitting negative results in reports and of two Nature papers on LLM limits in science.
Our analysis
The study’s most useful contribution is showing that detector failure is concentrated at a few generation boundaries rather than drifting gradually, which makes the problem manageable if screening is maintained like software, with retraining scheduled around major releases. arXiv’s caps and Hexagon’s disclosure tags show the publishing world working on the same problem from other angles: limiting volume, and making AI involvement explicit instead of trying to infer it. For editors, the safest position is that a detector trained before the latest model generation is not reliable evidence on its own.