Finetuning Forces Transformers to Actually Use Depth for Harder Reasoning

A common intuition about deep networks is that harder problems should recruit more of the network. For transformers, this would mean a 3-hop reasoning question travels further down the stack than a 1-hop lookup. A new interpretability study by Alicia Curth, Rachel Lawrence, Sushrut Karmalkar, and Niranjani Prasad puts that intuition under a microscope, and the picture that emerges is more nuanced than either the theory or the training folklore suggests.

The paper, accepted at the ICLR 2026 Workshop on Logical Reasoning of Large Language Models, asks a direct question: do transformers actually use their depth adaptively, spending more layers on harder inputs? Based on layer-wise probing of GPT-2, Pythia, Qwen, Llama, and Phi models, the answer is a qualified no for pretrained models and a much clearer yes once you finetune.

Why depth is the interesting axis

Recent theoretical work emphasizes depth as a critical bottleneck for transformers solving algorithmic and reasoning tasks of increasing difficulty. That framing motivates the insight that chain-of-thought approaches work because they equip models with additional effective computational depth, and it has spurred investigations into recurrent transformer architectures that let models adaptively determine their own depth. The result has been a wave of recurrent-depth designs like Universal Transformers, Huginn, and Ouro that loop the stack a variable number of times.

Sitting underneath all of that is an open question: do standard, non-recurrent transformers already do something like this implicitly? Even if depth use looks inefficient on average, models might have learned to reserve capacity for harder tasks. The authors define adaptive depth use as the property that the effective depth a model relies on to produce an output scales with task difficulty.