Skip to main content
Back

FABLE Forces AI to Actually Learn Facts, Not Just Parrot Them

A two-stage editing method injects atomic facts into shallow layers before touching deeper ones, closing the gap between text recall and fact recall.

Pro

  • FABLE is a two-stage editing method that anchors atomic facts in shallow Transformer layers before adjusting deeper layers for coherent text generation.
  • It targets a gap in prior methods like UnKE and AnyEdit, which recall edited passages holistically but fail on fine-grained fact questions.
  • Fine-grained Hit Rate on UnFine-UnKE with Llama-3-8B jumps from 23.69 (AnyEdit) to 53.31 with FABLE.
  • Introduces UnFine, a benchmark with Hit Rate and LCS Coverage metrics for fact-level evaluation of edits.
  • Preserves general capabilities on MMLU, MRPC, CoLA, RTE, NLI better than UnKE and AnyEdit.
  • Trade-off: about 10x slower per edit (~330s vs ~30s), and only tested at batch size 1. Code at github.com/caskcsg/FABLE.

Knowledge editing sounds simple in theory: instead of retraining a giant language model when a fact goes stale, you surgically rewrite the specific weights that hold that fact. It works reasonably well when the update is a clean triple like (Paris, capital_of, France). It breaks down when the update is a paragraph of messy prose, which is how roughly 80% of real-world knowledge is expressed. A new method called FABLE takes aim at a subtle failure mode in that unstructured setting: models can parrot the edited paragraph back verbatim, yet cannot answer questions about the facts inside it.

The recall trap in unstructured editing

Recent unstructured editing methods like UnKE and AnyEdit taught models to memorize whole passages by expanding the scope of editable parameters, using non-local block key-value storage instead of local layer storage, and swapping term-driven optimization for cause-driven optimization that edits the last token directly while preserving context. That approach hits high scores on holistic metrics like ROUGE-L and BERT-Score. According to the FABLE authors, those scores mostly capture surface-form mimicry.

Their diagnostic example involves a model edited with UnKE that can restate an entire biography paragraph about a fictional journalist when prompted for the person's background, yet fails when asked a pointed question about a specific detail buried in that same passage. The explanation the authors give: the model learned a high-level question-to-text mapping in the layers responsible for surface-form generation, without actually encoding the atomic facts into lower-level knowledge storage. Because Transformers flow information in one direction, later surface-form layers amplify whatever the earlier layers produced rather than correct it.