Library of Layers Lets Transformers Pick Their own Execution Path
A new transformer architecture keeps a shared pool of blocks and lets a router pick which one runs at each depth, mixing dense, looped, and mixture-of-experts behavior under one knob.
- Library of Layers (LoL) is a transformer that picks which block runs at each depth via a learned router.
- A single knob (library size) interpolates between dense, looped, and mixture-of-experts architectures.
- Strided window attention keeps per-token routing stable by shifting window boundaries at every depth.
- Forcing a routing path makes the model run a specific sudoku or maze algorithm with over 99.5% accuracy.
- Router outputs separate safe from unsafe text with up to 0.95 AUROC using a linear probe.
- Code is open source under MIT; no pretrained weights released.
Transformer language models usually send every token through the same fixed sequence of layers. Researchers behind Library of Layers (LoL) propose a different execution model: a router selects blocks from a shared library at each depth, allowing the order to change with the input.
By making execution order learnable, LoL gives one set of weights multiple processing paths. The architecture can reuse a small collection of blocks, select sparsely from a larger pool, or operate between those extremes. Its experiments also suggest that routing paths encode information about languages, algorithms, and text safety.
One parameter spans three familiar designs
A transformer block combines attention with feed-forward computation. Standard models assign a separate block to each position in a fixed stack; LoL stores blocks in a global library and asks the router which one should run at every step.
| Library size | Execution pattern | Closest architecture | Primary tradeoff |
|---|---|---|---|
| Small | Blocks are reused across many depths | Depth-recurrent transformer | Fewer parameters, more repeated computation |
| Near model depth | Blocks can specialize by depth | Dense transformer | Dedicated capacity for each processing stage |
| Large | The router selects sparsely from a broad pool | Mixture of experts | More parameters without activating all of them |
Changing the library size moves the model along this spectrum without requiring a separate architecture. A small library encourages repeated use, while a larger library gives the router more specialized blocks to choose from.
Conventional mixture-of-experts models usually preserve a fixed layer stack and route tokens among feed-forward experts inside each layer. LoL instead routes windows of tokens among complete transformer blocks drawn from one global pool.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
