Choosing a Local Open-Weight LLM for Clinical Note Annotation: Lessons From a German Hospital Study

A Scientific Reports study screened nine locally deployed open-weight LLMs to label symptoms in 1,100 German emergency-department notes. Results by symptom, why temperature 0.0 mattered, where a rule-based baseline held up, and a practical selection process for hospitals.

PUBLISHED // UPDATED
DURATION // 06 MIN READ
Hospital workstation with a desktop computer showing a clinical note and a local model terminal, cool overhead light, no logos
FIG.01 // VISUAL DOCUMENTATION Hospital workstation with a desktop computer showing a clinical note and a local model terminal, cool overhead light, no logos

Hospitals want to use language models to extract information from clinical notes, but sending patient records to a cloud API is often not acceptable. A study published in Scientific Reports on September 29, 2026 offers a practical alternative: a reproducible pipeline for choosing a locally deployed open-weight model for a specific annotation task, tested on real emergency-department notes from a German university hospital. The results show strong performance on some symptoms, no clear advantage on others, and one setting that mattered more than model size. This article summarises the method and turns it into guidance for teams doing similar work.

The study

“Development and validation of a pragmatic pipeline for clinical free-text annotation using locally deployed open-weight large language models” comes from a team at University Hospital Bonn. The goal was binary annotation: deciding whether each report mentions a patient having nausea, vomiting, diarrhoea or dysuria (painful urination).

The design:

  • 1,100 German emergency-department reports, from patients presenting with abdominal pain, manually annotated for the four symptoms.
  • 100 reports reserved for writing the prompt and testing temperature settings.
  • Nine open-weight models screened on a development set of 250 reports per symptom.
  • The best model per symptom tested on an independent validation set of 750 reports and compared with a negation-aware rule-based baseline, the kind of keyword system hospitals already use.

The prompt was deliberately simple and zero-shot. The aim was not to find the best possible model but to show that a screening workflow can pick one that is good enough for a given task.

Temperature 0.0 made the biggest difference

Before screening, the team tested how sampling temperature affected results. Most models were most stable at low temperature, and some degraded sharply as it increased:

ModelF1 at temperature 0.0F1 at 1.0Invalid responses at 1.0
mistral-large:123b0.9810.78930% (from 0%)
nemotron-mini:4b0.8630.7348% (from 0%)

The large Mistral model, one of the strongest in the study, produced invalid output in nearly a third of cases at temperature 1.0. The team used 0.0 for all later analyses. For yes/no labelling, randomness only adds errors; this setting is worth fixing before comparing models at all.

Which models won

The screening picked a different model for different symptoms:

  • Nausea: gemma2:27b (development F1 0.991, about 0.3 seconds per report)
  • Vomiting: mistral-large:123b
  • Diarrhoea and dysuria: llama3.3:70b

Mid-sized models either achieved the highest development score or overlapped with the top performer, while generally running much faster. GPU memory use reported by the Ollama runtime ranged from 2.63 GB to 112.70 GB across models, so model choice has real hardware implications.

Validation results against the rule-based baseline

On the independent 750-report sets:

SymptomLLM F1Rule-based F1Outcome
Vomiting0.9850.913LLM better
Nausea0.9790.724LLM better
Dysuria0.8240.705LLM ahead, but confidence interval includes zero
Diarrhoea0.8140.853No clear difference

For nausea and vomiting, the local models were clearly better. For diarrhoea and dysuria, the lower scores came from reduced positive predictive value, meaning more false positives, despite high specificity. Median inference time ranged from 0.298 to 1.653 seconds per report, fast enough for practical batch processing.

Why the errors matter

The authors analysed disagreements and found several causes: differences in how a symptom was operationally defined, symptoms that changed over time within a visit, inconsistent documentation, mentions the model missed, and five errors in the human reference labels themselves. Several of these are not model failures. They reflect how clinical notes are written and how “has diarrhoea” is defined. Any team doing this work needs written annotation criteria before evaluating models.

Hardware and privacy considerations

Running models locally keeps patient data inside the hospital network, which simplifies data protection reviews compared with sending notes to an external API. The trade-off is infrastructure. The study’s memory figures show that a 27-billion-parameter model and a 123-billion-parameter model have very different hardware needs, and that actual GPU memory use can exceed the quantised file size. Because mid-sized models matched or came close to the largest ones on most symptoms, starting with them is a sensible way to control cost.

Limitations

The study was single-centre, German-language, and limited to patients with abdominal pain and four binary symptoms. The authors stress that performance must be re-evaluated for each concept and documentation setting. The pipeline is the transferable part; the specific model winners may not be.

A separate study on retrieval-augmented vulnerability detection, reported by The Cryptonomist, made a similar point in a different field. Reproducing six open-source systems on a shared benchmark with open-weight models, the author found that published performance often did not transfer, and that even with ideal retrieval, pairwise accuracy stayed at 0.51. Results reported on proprietary models and private datasets need to be re-tested in your own setting.

A practical selection process for hospitals

Based on the study, a hospital team can follow these steps:

  1. Define each concept precisely, including edge cases such as historical symptoms or negated mentions.
  2. Annotate a local sample and split it into prompt-development, screening and validation sets.
  3. Fix temperature at 0.0 and require a strict output format, tracking invalid responses.
  4. Screen several open-weight models per concept, recording both accuracy and inference time.
  5. Validate the winner on held-out notes and compare with your existing rule-based system.
  6. Keep the rule-based system where it performs as well, since it is cheaper and easier to explain.
  7. Review disagreements, including checking whether the reference labels are wrong.
  8. Re-validate when documentation practices or models change.

For more on evaluation beyond leaderboards, see our guide to LLM benchmarking beyond the leaderboard, our look at small models like MiniCPM5-2B, and our analysis of open-weight versus closed model costs.

Our analysis

This study is valuable because it is modest. It does not claim one model is best; it shows how a hospital can choose a local model for a narrow task, validate it properly and accept that a rule-based system is sometimes just as good. The most transferable lessons are simple: fix temperature at zero for labelling, screen models per concept, measure speed as well as accuracy, and treat disagreements as a prompt to clarify definitions as much as to change models.