HySparse2 Cuts the Prefill Path for Tool-Using Agents

Researchers have described HySparse2, an attention architecture designed to reduce the cost of processing long tool responses. Tool-using language models often generate a short request, receive a long search result, document, or execution trace, and process that input before taking the next action. As the interaction history grows, this prefill work can consume more compute and memory than token generation.

HySparse2 combines YOCO-style cross-layer key-value sharing with sparse attention. According to the full paper, an 80-billion-parameter mixture-of-experts model using 3 billion parameters per token outperforms HySparse and a hybrid sliding-window baseline on long-context retrieval. The architecture also uses 2.92 times fewer prefill FLOPs than HySparse and 5.02 times fewer than the sliding-window configuration in the reported comparison.

Tool Responses Turn Prefill Into the Bottleneck

Prefill is the model pass that processes incoming tokens before generation begins. During that pass, each layer computes token representations and writes keys and values to the KV cache. The decoder later reuses that cache instead of recomputing the entire history for every generated token.

Agent workloads make this process unusually lopsided. A model may emit a short tool call and receive thousands of tokens in return, then repeat that pattern across several rounds. Each observation expands the history, increases attention work, and enlarges the cache that serving infrastructure must retain or transfer.

Existing efficiency methods reduce different parts of that cost:

• Sliding-window attention limits each token to nearby context, reducing compute while weakening access to distant evidence.
• Sparse attention selects a limited set of distant tokens, preserving broader retrieval while still running every model layer during prefill.
• YOCO-style sharing lets many decoder layers consume a shared KV cache, reducing cache duplication across the stack.

One Cache Plan, Two Sharing Layers

YOCO divides the transformer into a self-decoder followed by a cross-decoder. HySparse2 uses hybrid sliding-window attention in the self-decoder and hybrid sparse attention in the cross-decoder. In both halves, occasional full-attention layers provide global access while cheaper layers handle most of the sequence.