Skip to main content
Back

Peking University's LIFT Bakes Long Documents Into Model Weights, Beating RAG

A Peking University team proposes writing long documents directly into an LLM's weights via test-time fine-tuning on synthetic Q&A pairs.

Pro

  • LIFT fine-tunes long documents directly into LLM weights at test time instead of stuffing them into the context window.
  • A generator LLM converts each sentence into 5-10 synthetic QA pairs, which the target model trains on via LoRA.
  • Raw-text fine-tuning causes memorization; QA-format training produces genuine comprehension, the paper's central claim.
  • Perfect accuracy on Needle-In-A-Haystack up to 128K; LooGLE ShortQA jumps from 44.49 to 52.69 on Llama-3-8B.
  • Async producer-consumer pipeline keeps TTFT under 10 seconds for 8K inputs; general benchmarks like MMLU unaffected.
  • Code available at github.com/MuLabPKU/LIFT; weakness remains on cross-document reasoning tasks.

Long context remains the sore spot of modern LLMs. You can stretch the window, buy more GPUs to hold the KV cache, or bolt on a retriever, but none of those fully escape the quadratic cost of attention over a million tokens. A group from Peking University and BIGAI is proposing a different trade-off: stop trying to fit the document in the prompt, and instead fine-tune it into the model's weights at inference time.

Their framework, called Long Input Fine-Tuning (LIFT), turns each long input into a short burst of LoRA training before the model ever sees the user's question. Rather than plain continued pretraining on new long contexts, LIFT uses carefully designed synthetic tasks, generated by another LLM, to push comprehension beyond memorization. The result is a short-context model that can answer questions about a document without the document being in its context window at all.

Why raw text fine-tuning falls flat

The obvious version of this idea, take a long document and just continue-pretrain the model on it, has been tried before under names like TempLoRA and TTT-E2E. It mostly fails. Fine-tuning on raw text produces rote memorization rather than comprehension, and the authors show this concretely on SQuAD: a model fine-tuned on raw context tends to echo back lexically similar phrases from the passage while missing the actual meaning.

Reformatting the document before training on it fixes much of this. Raw text often expresses knowledge in a descriptive, implicit, compact form, while QA pairs turn that knowledge into explicit mappings from questions to answers, which is easier for models to internalize. That matches an old finding from human learning research: writing your own questions while reading is one of the most effective study strategies.