Skip to main content
Back

Johns Hopkins' IA2 Makes Fine-Tuning Think Like a Prompt on 4 Examples

Researchers at Johns Hopkins show that priming a fine-tuning run with in-context learning activations lifts accuracy and calibration across 12 benchmarks.

Pro

Paper, Full Access
  • IA2 is a self-distillation priming step that aligns SFT model activations with those of the same model doing ICL.
  • Authors show ICL and SFT produce fundamentally different activation patterns despite similar surface outputs (paper).
  • Priming with IA2 before SFT beats plain SFT on accuracy and calibration across 12 benchmarks and two model families.
  • Over 13,000 models trained on Qwen3-4B and Llama-3.2 1B/3B with rank-8 LoRA on Q, K, O projections.
  • SVD analysis shows IA2 finds weight subspaces that are orthogonal to and unreachable by SFT-only training.
  • Weakness: extreme ICL alignment can inherit ICL overfitting; long-form generation needs higher LoRA ranks.

Supervised fine-tuning and in-context learning look like two paths to the same destination: get a base model to behave differently on your task. A new paper from Johns Hopkins argues they are not the same route at all, and that the internal wiring one produces can be used to rescue the other. The method, called IA2 (ICL Activation Alignment), is a self-distillation step you run before standard SFT, and it consistently improves both accuracy and calibration in low-data regimes.

The starting point is a mechanistic observation. ICL and SFT produce distinct activation patterns, indicating that the two methods achieve adaptation through different functional mechanisms. Even when you train a fine-tuned model on the exact same demonstrations you would put in an ICL prompt, the hidden-state fingerprints diverge, especially in the middle layers where the model would be doing the abstract, cross-example reasoning that makes in-context learning generalize.

Heatmap comparing layerwise activation similarity with ICL across SFT only, IA2 only, and IA2 to SFT for Qwen-4B, Llama-3B, and Llama-1B

Why SFT keeps losing to a good prompt

In the few-shot setting that most practitioners actually live in, ICL tends to be better calibrated than SFT and often more accurate too. The authors attribute this to the training signal itself: SFT only sees output tokens, so it can latch onto shortcuts that fit the tiny training set but fail on new inputs. ICL, by contrast, has to run generalizable circuits at inference time because a single set of weights must handle many different prompts. The cost, of course, is that every query eats context tokens for demonstrations, which gets expensive at scale.

Prior work in context distillation tried to bake ICL behavior into weights, but only by training the student to reproduce the teacher's output tokens. The Hopkins team's argument is that matching outputs does not mean matching function. Getting the fine-tuned model to think like an ICL model requires aligning the activations, not just the answers.