Loyola's TLA-Prover Beats Untuned AI at Formal Verification by 3.5x
TLA-Prover fine-tunes a 20B model with RL to generate verified TLA+ specs, hitting 30% pass@1 -- 3.5x better than any untuned LLM baseline
Takeaways
- 3.5x improvement: TLA-Prover hits 30% pass@1 on TLC model-checking, up from the 8.6% best untuned LLM baseline across 25 models.
- Two-stage training: Supervised fine-tuning on verified specs, followed by repair-based GRPO reinforcement learning using TLC as the reward signal -- no human labels needed.
- Diamond tier evaluation: A mutation-testing trick automatically alters correctness properties to check they are falsifiable, preventing models from gaming benchmarks with trivially-true assertions.
- LoRA on a 20B MoE model: Uses rank-8 LoRA adapters on attention layers of
gpt-oss-20b; MoE expert weights are not fine-tuned due to architectural constraints. - DPO underperforms GRPO: A DPO variant from the same SFT checkpoint reaches only 20% Diamond vs. 30% for the GRPO model, confirming the value of the repair loop.
- No public release yet: The paper does not announce a public checkpoint or code repo at time of publication.
TLA-Prover is a new 20-billion-parameter model that can generate formally verified TLA+ specifications from natural-language descriptions. Researchers at Loyola University Chicago trained it using a combination of supervised fine-tuning and reinforcement learning, with the TLC model checker acting as the judge. The result: a 30% pass rate on a held-out benchmark, compared to just 8.6% for the best untuned LLM baseline -- roughly a 3.5x improvement.
Why TLA+ is hard to automate
TLA+ (Temporal Logic of Actions) is a formal specification language created by Leslie Lamport. It is used for designing, modelling, and verifying programs, especially concurrent and distributed systems. Think of it as mathematically rigorous pseudocode: you describe what a system is supposed to do, and a tool called the TLC model checker exhaustively explores every possible execution path to find violations.
Amazon Web Services has used TLA+ since 2011, and TLC model checking uncovered bugs in DynamoDB, S3, EBS, and an internal distributed lock manager -- some bugs required state traces of 35 steps. Microsoft Azure used TLA+ to design Cosmos DB, a globally-distributed database with five different consistency models. The language is powerful, but it comes with a steep learning curve.
Its steep learning curve and the meticulous nature of crafting specifications have historically limited its widespread adoption. Now, LLMs are entering this domain, promising to democratize formal verification. The problem is that LLMs are notoriously bad at it. LLMs frequently produce TLA+ specifications that fail the TLC model checker for semantic reasons. Getting the syntax right is one thing; getting the logic right -- so that TLC can actually verify safety and liveness properties -- is much harder.
The baseline is brutally low
A prior study called FormaLLM provides the first systematic evaluation of LLM-based TLA+ generation, testing 30 models across eight families under four prompting strategies. Their headline finding of 26.6% SANY parse and 8.6% TLC model-check averaged across 25 open-weight models establishes the untuned-LLM floor. In other words: the best off-the-shelf model can barely get a quarter of its outputs to parse correctly, and fewer than 1 in 10 actually pass the model checker. FormaLLM is an evaluation study with no task-specific training, and the authors explicitly identify task-trained models as the natural next step. This paper answers their call directly.