INCRT Grows Transformer Attention Heads on Demand, Beating BERT With 7x Fewer Parameters
A new Transformer variant grows its own attention heads based on task geometry, matching BERT-base with 3-7x fewer parameters and no pre-training.
ProTakeaways
- INCRT grows its own attention heads during training, starting from one head and adding based on task geometry.
- Two theorems prove the process terminates at a minimal-sufficient architecture and bound the final head count.
- Predicted and observed head counts match within 12% on CoV-2 and SST-2 benchmarks.
- Matches or exceeds BERT-base on genomic tasks with 3-7x fewer parameters, no pre-training.
- Growth-pruning duality lets INCRT detect task drift and reorganize heads without external signals.
- Single-layer only so far; multi-layer depth growth is theoretically defined but not validated.
Every Transformer ever trained has a dirty secret: most of its attention heads are useless. A new paper from Giansalvo Cirrincione at Université de Picardie Jules Verne proposes an alternative. Instead of picking head counts by trial and error and pruning later, INCRT (Incremental Transformer) starts with a single attention head and grows the architecture during training, driven entirely by the geometry of the task.
The redundancy problem
The starting observation is well-known but rarely acted on. The paper notes that between 50% and 80% of the attention heads in trained Transformer models can be removed without any measurable loss of performance. That points to a structural flaw. The number of heads, the depth, and the head size are all fixed as hyperparameters before training, with no mathematical link to what the task actually needs.
The root cause lives inside the attention weight product itself. The matrix M = W_Q W_K^T simultaneously encodes two geometrically opposite functions: a symmetric part governing which tokens attend to each other reciprocally, and an antisymmetric part governing the direction of information flow. Because the architecture imposes no separation between these two roles, the learning algorithm must discover the decomposition implicitly, at the cost of structural redundancy. Standard practice, then, is to train a bloated model and prune afterward. That guarantees minimality but never sufficiency, so pruning can silently remove capacity the task genuinely needs.
Growing an architecture from a scalar
INCRT flips the direction. It maintains a running quantity called the residual matrix, a measure of directional structure in the data that the current heads have not yet captured. The largest eigenvalue of that matrix, computed online during the normal forward pass, tells the system whether it is capacity-starved. If so, a new head is added in the direction that most reduces the measured deficit. When no deficit remains above a prescribed threshold, the system stops. The final architecture is both minimal and sufficient by construction.