Kyutai's Perturb-Select-Vote Beats Neural Thickets With a Single Bias Vector

Neural Thickets reports large reasoning gains from adding Gaussian noise to a pretrained language model’s weights, ranking the resulting copies on a small labeled set, and taking a majority vote. Researchers at Kyutai and collaborating institutions reproduced those gains, then matched or exceeded them by perturbing a single vector added to the model’s output scores. The finding replaces billions of noisy parameters with one vocabulary-sized bias vector per ensemble member.

The authors formalize the recipe as Perturb-Select-Vote (PSV): create varied model outputs, select members using a validation set, and aggregate their answers by vote. Neural Thickets generates many noisy model copies, evaluates them on 200 labeled examples, retains the best 50, and uses their majority answer at test time. PSV lets the researchers change the source of variation while preserving selection and voting.

One Bias Vector Replaces Weight Noise

On Countdown with OLMo-3-7B-Instruct, the unmodified model scores 66.6%, while full-weight PSV reaches 90.8%. Perturbing any single decoder block improves on the baseline, and perturbing only the final block reaches 91.8%. Those results show that coordinated changes across the full network are unnecessary for the reported gain.

The simplest version freezes every model parameter and perturbs only the logits, the raw scores assigned to possible next tokens before probabilities are calculated. Each ensemble member samples one Gaussian vector with one value per vocabulary token, then adds that same vector at every decoding step and for every prompt. The vector behaves like a fixed output-layer bias.