RILKE Rewrites AI Facts 1,000 Times Without Breaking the Model
RILKE edits LLM knowledge by steering hidden states instead of rewriting weights, staying stable across 1,000+ sequential edits where prior methods collapse.
Takeaways
- RILKE edits LLMs by intervening in hidden representations, not weights, keeping the base model frozen.
- Maintains BertScore 1.000 after 1,000 sequential edits on UnKE where UnKE and AnyEdit collapse to 0.01.
- Preserves MMLU at 0.622 versus 0.633 base, while WISE drops to 0.584 and UnKE to 0.126.
- Shared-subspace clustering cuts storage to 29.4 MiB versus WISE at 224 MiB for 1,000 edits.
- Router correctly dispatches over 93% of paraphrases and filters over 98% of unrelated MMLU queries.
- Compatible with EasyEdit; tested on Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct.
Every deployed language model eventually goes stale. Facts change, policies shift, and the model keeps confidently emitting the version it learned during pretraining. Retraining is expensive, retrieval-augmented generation fights with parametric memory, and the family of methods known as knowledge editing has a nasty habit of breaking the model after a few dozen edits. A new paper introduces RILKE, a method that sidesteps weight surgery entirely and instead steers the model's hidden representations, holding up across thousands of sequential edits while keeping general capabilities intact.
Why knowledge editing keeps collapsing
The core failure mode of existing editors is called edit collapse. Parametric approaches that modify weights directly suffer from it: as new knowledge accumulates, performance degrades until the model becomes unusable. Free-form, unstructured knowledge that cannot be reduced to simple subject-relation-object triples takes the worst beating.
The paper's case studies show what this looks like in practice. After 1,000 sequential edits on the UnKE benchmark, methods like UnKE and AnyEdit degenerate into token soup, producing outputs like repeated "a a a a" sequences or fragments of unrelated URLs. Memory-based approaches like WISE stay coherent but drift semantically, capturing the first few tokens correctly before wandering off topic. Fine-tuning a single sub-module such as an extra feedforward layer cannot cleanly hold hundreds of distinct, complex facts without them bleeding into each other.
The representation space bet
RILKE (Representation Intervention for Lifelong KnowledgE Control) builds on Representation Fine-Tuning, a technique that leaves model weights frozen and instead learns a small module to nudge intermediate hidden states toward a target. The novelty here is showing that this substrate has two geometric properties that make it uniquely suited to lifelong editing: