Skip to main content
Back

RAMPART Lets Smaller Models Beat Larger Ones by Controlling Context Order

RAMPART treats LLM context assembly like a compiler pass, using five primitives to control what goes in, where, and when — with measurable gains across five models.

Paper, Full Access
  • RAMPART is a compile-time memory model for LLM agents that treats context assembly as a programmable operation over a named block registry.
  • Five primitives (promote, gate, write, evict, rollback) control what enters the prompt and where, at zero token cost.
  • Task success collapses past block position 7-12; grouping critical blocks with semantic neighbours lifts pass rates by tens of percentage points.
  • Block grouping raises Mistral-7B's pass rate ~5x at the hardest registry size; a smaller model with the intervention can beat a larger model without it.
  • Relevance gating cuts prompt cost by 67.8% while recovering 83% of full-context performance; schema eviction guarantees 0% tool invocations.
  • The effect is replicated across Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3, and Qwen3-14B — paper on arXiv.

Every LLM agent has a dirty secret: it doesn't actually control what goes into its own context window. Tools dump outputs, RAG pipelines inject retrieved chunks, system prompts pile up , and the model just gets whatever lands in the buffer. A new paper from Nikodem Tomczak proposes RAMPART, a compile-time memory model that treats context assembly as a first-class, programmable operation rather than an afterthought.

The problem hiding in plain sight

LLM agents increasingly operate in settings where a single context window is far too small to capture everything relevant. Memory , the ability to persist, organize, and selectively recall information across interactions , is what turns a stateless text generator into a genuinely adaptive agent. But most frameworks still treat context construction as a passive process: append what you have and hope the model figures it out.

The deeper issue is positional. The "lost-in-the-middle" effect is a demonstrated phenomenon where LLMs perform significantly worse when relevant information is placed in the middle of the input context rather than at the beginning or end. LLMs exhibit a U-shaped attention bias where the tokens at the beginning and end of their input receive higher attention, regardless of their relevance. This isn't a quirk of one model family , it is not a minor edge case; it affects every production LLM.

Treating the LLM context window like persistent storage causes most agent failures. Most production agent failures are not model failures but memory architecture failures , and the fix requires managing the context window the way an operating system manages RAM: with deliberate control over what goes in, what stays, and what gets evicted.

RAMPART: context as a compiled artifact

RAMPART is a compile-time memory model and pure in-RAM block registry for LLM-based agents. Context assembly is a programmable runtime operation where content is compiled from a structured registry under explicit policy for ordering, inclusion, and eviction. The key mental model: instead of building a prompt by concatenation, you maintain a named registry of content blocks and then