Cite Pretrain Teaches AI to Remember Sources Without Searching the Web
A new training recipe called Active Indexing teaches LLMs to cite the documents they learned from, no retriever required at inference time.
- New paper Cite Pretrain teaches LLMs to cite sources without a retriever at inference time.
- Two stages: continual pretraining binds facts to document IDs, then instruction tuning elicits citations.
- Active Indexing augments training with paraphrased facts and bidirectional source-to-fact and fact-to-source objectives.
- Beats Passive Indexing baseline by up to 30.2% citation precision on Qwen-2.5-7B and 3B.
- Introduces CitePretrainBench, mixing Wikipedia, Common Crawl, arXiv, and novel documents for short and long-form tasks.
- Internal citations complement external retrieval, winning under noisy retrieval and improving hybrid pipelines.
Cite Pretrain teaches language models to cite from memory
Researchers have released Cite Pretrain, a training method that teaches language models to associate facts with persistent document identifiers. After training, a model can answer questions and cite documents from its training corpus without querying an external retriever during inference.
Yukun Huang, Sanxing Chen, Jian Pei, Manzil Zaheer, and Bhuwan Dhingra describe the method in their paper. Their approach moves part of the citation pipeline from inference into continual pretraining, reducing retrieval latency and dependence on a live search index. Its citations remain limited to sources indexed during training, and each generated identifier must resolve through a maintained document catalog.
Why citation systems use retrieval
Standalone language models can generate plausible but unsupported titles, URLs, and references because standard pretraining does not explicitly preserve a dependable link between a fact and its source. Production systems commonly address this problem with retrieval-augmented generation, or RAG, which searches an indexed corpus, inserts relevant passages into the prompt, and asks the model to answer from that evidence.
RAG adds an index, a retrieval request, and document-selection errors to every query. Poorly ranked passages can weaken both the answer and its citations, while maintaining embeddings and retrieval infrastructure adds operational cost. Cite Pretrain instead encodes fact-to-source associations in the model’s weights before deployment.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
