Nature Medicine Study: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 Beat OpenEvidence and UpToDate Expert AI, and Why the Result Is Disputed

A June 2026 Nature Medicine study found three frontier LLMs outscored two specialized clinical AI tools on MedQA, HealthBench and real clinician queries. The results, the vendor pushback including a retraction demand, and what it means for enterprises choosing general models versus specialist AI.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
Nature Medicine Study: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 Beat OpenEvidence and UpToDate Expert AI, and Why the Result Is Disputed
FIG.01 // VISUAL DOCUMENTATION Nature Medicine Study: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6 Beat OpenEvidence and UpToDate Expert AI, and Why the Result Is Disputed

A June 2026 Brief Communication in Nature Medicine found that three general-purpose frontier models, GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6, outperformed two specialized clinical AI tools, OpenEvidence and UpToDate Expert AI, in all three of its evaluations (Nature Medicine). On 500 MedQA questions, Gemini 3.1 Pro scored 97.4% against 89.6% for OpenEvidence and 88.4% for UpToDate Expert AI. The study is contested: OpenEvidence has demanded a retraction and Wolters Kluwer, which makes UpToDate, disputes the methodology. Its most durable lesson for enterprises is not that general models always win, but that specialist AI products should be independently tested against general models on real workloads before anyone pays a premium for them.

What the study measured

The authors evaluated the systems in three stages:

StageWhat it testsSize
MedQAMedical licensing-style knowledge questions500 questions
HealthBenchAlignment of free-text answers with physician-written rubrics500 items
Real clinical queries (RCQ)Answers to de-identified questions physicians actually asked a general-purpose model in a live clinical setting, rated blind by clinicians100 queries; 12 US clinicians; 1,800 annotations

Google Search’s AI Overview was also included on the RCQ benchmark because clinicians routinely encounter it. The paper reports that the clinical AI tools performed comparably to AI Overview on that benchmark, and concludes that the findings “highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.”

The headline numbers

MedQA accuracy, as reported by heise online and Bursera Consulting’s analysis of the paper:

SystemTypeMedQA accuracy
Gemini 3.1 ProGeneral-purpose97.4%
GPT-5.2General-purpose94.2%
Claude Opus 4.6General-purpose90.2%
OpenEvidenceSpecialized clinical89.6%
UpToDate Expert AISpecialized clinical88.4%

The general models also scored higher on HealthBench and on the clinician-rated real queries. heise reports that UpToDate Expert AI refused 19% of the RCQ queries, a figure that became central to the dispute.

Why the result is disputed

Within weeks the study drew a sharp response, summarised by Bursera Consulting from reporting by Becker’s Hospital Review:

  • OpenEvidence publicly alleged methodological flaws and an undisclosed conflict of interest on June 13, and on June 15 wrote to Nature Medicine demanding retraction, a public apology and an independent review. Its central argument is contamination: MedQA and HealthBench are public benchmarks that frontier models may have seen in training. The company also backed a counter-study.
  • Wolters Kluwer did not demand a retraction but disputed the measurement. Peter Bonis, chief medical officer of Wolters Kluwer Health, said the study “confused clinical quality and complete-sounding answers,” and argued that counting refusals as failures ignores whether declining to answer was the safer choice.
  • Nature Medicine’s article page shows that a Matters Arising response was published on September 3, 2026, meaning the debate has continued in the journal itself.

These objections matter. Public benchmarks reward test-taking, and a clinical tool that refuses an ambiguous question may be behaving responsibly. The RCQ benchmark, built for the study from real physician queries and blind-rated by clinicians, is the part least exposed to contamination, and it still favoured the general models, though the vendors contest how quality was judged there too.

Why general models can beat specialist tools

Even with the caveats, the result fits a broader pattern:

  • Breadth of training. Frontier models are trained on large volumes of medical literature alongside everything else, and clinical questions often cut across pharmacology, physiology and patient context.
  • Reasoning ability. Specialist products built on top of LLMs inherit the reasoning ability of whatever model they use, and may lag the newest frontier releases.
  • Release speed. Frontier labs ship new models frequently; products that wrap an older model, add retrieval and pass internal validation can fall behind between upgrades.

Specialist tools still sell things a benchmark does not measure: curated, licensed evidence (OpenEvidence licenses major journals; UpToDate is written by physician contributors and editors, according to Wolters Kluwer), citations, editorial accountability, workflow integration and liability arrangements.

What this means for enterprise AI decisions

DecisionImplication
Buy a specialist tool vs. build on a general modelBenchmark both on your own tasks; a general model plus retrieval over your data may match or beat a niche product
Regulated workflowsSpecialists’ advantages are often audit trails, sourcing and accountability, not raw accuracy; a hybrid of general model plus compliance layer may work best
Vendor claimsAsk for independent evaluations, not only vendor benchmarks
Refusals and safetyMeasure how often a system declines and whether those refusals are appropriate, not just accuracy
Benchmark contaminationPrefer private, task-specific test sets built from your real queries

Test at scale, on real workloads

Benchmarks describe ideal conditions. In an opinion piece for RCR Wireless News, Sashi Jeyaretnam of VIAVI, a testing vendor, argues that customer-facing AI must be validated for performance, security and cost under realistic load. Latency metrics such as time to first token and time to last token “degrade non-linearly under load,” so teams should measure P95 and P99 tail latency rather than medians. The author also states that inference typically accounts for 80–90% of compute spend over a model’s production lifecycle, which makes choosing the right model size an economic decision as much as a quality one.

For more on why leaderboard scores can mislead, see our guide to LLM benchmarking beyond the leaderboard. For how fast frontier models are moving, see our coverage of Anthropic’s Claude Mythos release.

A practical evaluation checklist

  1. Build a private test set from real, anonymised queries in your domain.
  2. Include general and specialist options and the current frontier models, not last year’s.
  3. Use blinded expert review for quality, with explicit criteria for correctness, completeness, safety and clarity.
  4. Track refusals and hallucinations separately and judge whether refusals were appropriate.
  5. Test under realistic load for latency and cost.
  6. Re-run regularly, because rankings change with each model release.

Our analysis

The Nature Medicine study is an important data point, not a final verdict. Its MedQA and HealthBench results are vulnerable to the contamination critique, and the vendors’ objections about refusals and answer quality deserve a hearing. But the core finding, that expensive specialist AI products did not beat frontier general models in an independent evaluation that included real clinician questions, should change how organisations buy AI in any field. Specialist vendors now have to prove that their curated data, citations, safety behaviour and integration justify their cost, and buyers should insist on head-to-head testing on their own workloads before assuming that domain-specific means better.