Skip to main content
Back

SJTU's Depth-Attention Beats Vanilla Transformers by 2.3 Points With Zero Extra Cache

Depth-Attention lets Transformer layers selectively reuse value states from earlier layers inside attention itself, adding zero KV cache overhead while improving accuracy by up to 2.3 points

  • What it is: Depth-Attention lets each Transformer layer use its own query to selectively blend value states from earlier layers before running standard self-attention.
  • Zero KV cache overhead: Depth-mixed values overwrite the standard V-cache slot, so inference memory is identical to a vanilla decoder -- unlike DenseFormer (24 GiB extra at 128K) or mHC (1.5 GiB extra).
  • Accuracy gains: Up to +2.3 accuracy points over vanilla Transformer at 3B scale; beats all cross-layer baselines (mHC, Attention Residuals, DenseFormer) on average downstream accuracy.
  • Negligible compute: +0.004% arithmetic FLOPs and only +1.18% inference latency; training overhead is ~10% due to unfused kernels, far below baselines at 50-321%.
  • Scales consistently: Benefits hold from 360M to 3B parameters and extend to looped (recurrent) Transformers.
  • Code available: Official implementation released at github.com/LUMIA-Group/Depth-Attention from Shanghai Jiao Tong University's LUMIA Lab.

Every Transformer layer can look anywhere in the sequence to pick out what it needs. But when it comes to looking back through the model's own depth, layers are stuck with a blunt instrument: the residual stream, which collapses everything from all previous layers into a single running sum. A new paper from LUMIA Lab at Shanghai Jiao Tong University proposes Depth-Attention, a mechanism that gives each layer the same selective, content-based access across depth that self-attention already provides across the sequence.

The blind spot in every Transformer

Self-attention selects information freely across the sequence, but across depth, Transformers merely add each layer's output to the residual stream, so later layers cannot selectively reuse earlier-layer representations. Think of it this way: layer 40 of a 48-layer model can query any token in the context window, but it cannot ask "what did layer 10 think about this token?" It only sees the accumulated sum of everything that happened before.

A wave of recent work has tried to fix this. DenseFormer aggregates the outputs of preceding layers through learned depth-wise averaging, and Hyper-Connections replace the single residual path with multiple interacting streams that exchange information across depths; most recently, Attention Residuals make depth-wise selection adaptive by replacing fixed residual accumulation with a softmax attention over preceding layer outputs. The catch? These mechanisms all operate on hidden states, which lie outside the key-value cache. At inference, a model must therefore maintain or recompute this extra state on top of its key-value cache -- an overhead that grows in relative terms as modern LLMs aggressively shrink the cache with grouped-query attention (GQA) or multi-head latent attention (MLA).

Attention, but across layers instead of tokens

Depth-Attention sidesteps the memory problem with a conceptually clean move: run the same attention operation, but along the depth dimension instead of the sequence dimension. Before a layer attends over the sequence, its query attends along depth over the keys of earlier layers at the same token position and mixes their values into the value that self-attention then reads.