Sichuan University's AdapShot Beats Fixed Shot Counts With 4.64x Speedup
AdapShot dynamically picks how many examples to show an LLM per query, cutting inference time 4.64x while boosting accuracy 10% over the previous best method.
- AdapShot dynamically adjusts the number of in-context examples per query using output entropy as a confidence signal, rather than using a fixed shot count.
- It achieves a 4.64x speedup and ~10% accuracy gain over DBSA, the previous state-of-the-art efficient many-shot ICL method.
- A Semantically-Aware KV Cache (SAKVC) pre-encodes all demonstrations offline so the model never re-processes examples redundantly.
- A Position Decoupling and Re-encoding (PDR) module exploits RoPE's rotational math to fix positional encodings when KV pairs are reordered; removing it drops GSM8K accuracy by 29.7%.
- Evaluated on 7 tasks (CoLA, QNLI, PIQA, SQuAD v2, SVAMP, GSM8K, MathQA) across Qwen2.5 7B, Llama3.2 3B, and Llama3.1 8B models.
- No code release yet; paper available at arxiv.org/abs/2605.03644. Works only with RoPE-based models.
Most people using in-context learning (ICL) pick a shot count and stick with it. Ten examples for every query, or a hundred, or five hundred. AdapShot, a new paper from researchers at Sichuan University, argues that this one-size-fits-all approach is fundamentally broken -- and fixes it with a system that decides, on the fly, exactly how many examples each individual query actually needs.
The result is striking: AdapShot achieves an average performance gain of around 10% and a 4.64x speedup compared to state-of-the-art DBSA, the previous leading method for efficient many-shot ICL.
Why fixed shot counts are a bad idea
In-context learning lets you adapt a language model to a task by prepending examples directly into the prompt -- no fine-tuning required. LLMs have demonstrated remarkable ICL capabilities, enabling models to adapt to downstream tasks without parameter updates. While traditional few-shot ICL typically relies on fewer than 10 examples, the latest research paradigm is gradually evolving toward Many-Shot ICL. Studies show that extending the context window to include hundreds or even thousands of examples can significantly unlock the model's potential in complex reasoning and knowledge-intensive tasks.
But scaling up examples blindly creates two compounding problems. First, existing methods typically rely on a predetermined, fixed number of shots. This static approach often fails to adapt to the varying difficulty of different queries, leading to either insufficient context or interference from noise. A simple arithmetic question does not need 512 examples; a nuanced reading comprehension task might. Padding every query to the same length wastes compute on easy queries and potentially hurts performance by drowning the signal in irrelevant demonstrations.
Second, the prohibitive computational and memory costs of long contexts severely limit many-shot's feasibility. The key-value (KV) cache can consume up to 70% of total memory during inference for long-context workloads, and attention cost grows quadratically with sequence length.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
