Norwegian Hospitals Built KliniskVestBERT on 16M Patient Records and Cut Fine-Tuning by 8x
Norwegian hospitals trained three BERT models on 16 million real clinical records, beating general-purpose baselines on every medical NLP task tested
Takeaways
- KliniskVestBERT is a suite of three BERT models continued pre-trained on 16.2 million de-identified Norwegian clinical records from Helse Vest hospitals.
- All three models beat their general-purpose baselines on every clinical NLP benchmark tested, including real-world stroke detection and drug extraction tasks.
- The training corpus spans 5.1 billion tokens across 715 document types including discharge summaries, surgical reports, and nursing notes in both bokmål and nynorsk.
- A custom de-identification pipeline achieved 98% sensitivity, using a "hidden-in-plain-sight" method that replaces real values with plausible fakes rather than redaction tokens.
- Clinically adapted models reach peak fine-tuning performance in 1-2 epochs vs. 8-10 epochs for baseline models, cutting downstream compute by up to 8x.
- Models are not yet publicly released; the paper serves as a blueprint for hospital-scale clinical NLP adaptation in low-resource languages.
Clinical language is a beast of its own. Doctors write in shorthand. Nurses use informal phrasing. Surgical reports are dense with domain-specific terminology that never appears in Wikipedia or news corpora. General-purpose language models, even large ones, stumble badly in this environment. KliniskVestBERT is a direct answer to that problem, and it comes from an unlikely source: a consortium of Norwegian regional hospitals.
Researchers from Helse Vest ICT, working with all four hospital entities in Norway's Vestland region (Helse Bergen, Helse Fonna, Helse Førde, and Helse Stavanger) and healthcare IT vendor DIPS, have released a suite of three clinically specialized BERT encoder models. Each one was built by continuing the pre-training of an existing Norwegian language model on a massive corpus of real, de-identified patient records.
The data problem that makes clinical NLP so hard
The increasing application of NLP in healthcare demands language models specifically attuned to the complexities of clinical language. The core difficulty is that medical texts are nothing like the data most models train on. Models trained solely on publicly available medical resources such as PubMed abstracts or medical textbooks may not adequately represent the language patterns found in patient records, which are often characterized by less formal writing styles, abbreviations, and a greater degree of contextual specificity.
For Norwegian specifically, the challenge is compounded. Prior work has focused primarily on English and other high-resource languages, leaving a gap in Norwegian-specific clinical NLP research. Norwegian also has two official written standards, bokmål and nynorsk, both of which appear in clinical documentation and must be handled correctly.
16 million documents, carefully scrubbed
The training dataset is based on all clinically relevant text documents from almost all patients of the hospitals of Helse Vest in the years 2019, 2021, and 2022. The team excluded data from 2020 because of concerns regarding a COVID-19 bias. The result is a dataset that is genuinely representative of a regional hospital population, covering all medical specialties including mental health and all age groups.