Shanghai Jiao Tong's LatentSkill Cuts Agent Token Use by 72% Without Prompt Injection
LatentSkill converts text-based agent skills into LoRA adapters via a hypernetwork, cutting token overhead by up to 72% while boosting task success rates by over 21 points
Takeaways
- LatentSkill converts text-based agent skills into LoRA adapters via a hypernetwork, eliminating the need to paste skill text into every prompt step.
- On ALFWorld, it improves task success by 21.4 points on seen tasks and 13.4 on unseen, while using 64.1% fewer prefill tokens than in-context skill prompting.
- On Search-QA, it achieves a 3.0-point exact match improvement with 72.2% lower skill-token overhead.
- Generated LoRA adapters form a structured semantic geometry in weight space, with skills from different domains forming separable clusters even for out-of-distribution inputs.
- Skills can be composed via parameter-space arithmetic, but only when decomposed into semantically aligned sub-components first; naive weight averaging fails.
- Under prompt hijacking attacks, in-context skill collapses from 52.9% to 8.57% success, while LatentSkill retains 38.6%; code is open-source on GitHub.
Every time an LLM agent takes a step in a task, it typically re-reads the same skill documentation in its prompt. A skill that tells the agent how to pick and place objects gets pasted into the context window at every single decision step, over and over. A new paper from Shanghai Jiao Tong University and collaborators asks a simple but powerful question: what if the skill lived in the model's weights instead of its prompt?

The prompt tax on agentic systems
A common design retrieves relevant skills from a skill library and inserts them into the prompt when the agent selects an action. This works, but it does not scale. The same skill text may be inserted repeatedly across decision steps, consuming context and increasing prefill cost; long inputs also make it harder for models to use all supplied information robustly. On top of the efficiency hit, there is a security problem: skills kept as readable prompt content can expose proprietary procedures and share the instruction channel with potentially untrusted observations.
The alternative, fine-tuning skills directly into the model, trades one problem for another. Parametric alternatives, such as agent fine-tuning or curriculum learning, avoid inserting skill text at inference time, but they fuse skills into the model parameters. As a result, individual skills become difficult to update, remove, or combine. This is the trilemma the paper sets out to break: avoid repeated skill text, but keep skills modular and composable.
Skills as weights, not words
LatentSkill is a framework that converts textual skills into plug-and-play LoRA adapters through a pretrained hypernetwork. A hypernetwork is a neural network that generates the weights of another network. Here, it takes a text skill document as input and outputs a LoRA adapter (a small set of low-rank weight matrices) in a single forward pass. Given a skill description, a trained hypernetwork generates a skill-specific LoRA adapter in a single forward pass, which is then mounted on the backbone LLM during inference. The original skill text is no longer included in the prompt, reducing both context cost and exposure as readable text.
The generated adapters remain modular: they can be loaded, unloaded, replaced, or scaled without retraining the backbone. This is the key insight: you get the efficiency of baked-in skills, but the flexibility of plug-in modules.