Hidden Math Bug Silently Corrupts AI Training Gradients Near Equal Distributions
A new arXiv preprint extends recent softmax findings to KL and JSD divergences, showing standard code silently corrupts gradients when distributions are close.
- New arXiv preprint by Brandon Amos extends softmax numerical-bug findings to KL, JSD, and entropy.
- Standard divergence code loses precision via cancellation when two distributions are nearly equal.
- Affects knowledge distillation, KL-regularized RL (PPO, GRPO), and proximal optimization methods.
- Also covers Monte Carlo KL estimators k1, k2, k3 used in policy gradient training.
- Proposes stable reformulations with example implementations, validated via property-based tests.
- Builds on Barron et al. (2026) softmax and cross-entropy fix from Google DeepMind.
Brandon Amos’s short arXiv preprint extends recent work on numerical failures in softmax and cross-entropy. It finds related floating-point problems in forward and reverse Kullback-Leibler divergence, generalized Jensen-Shannon divergence, Monte Carlo KL estimators, and categorical entropy. The paper also provides stable reformulations and example code.
A softmax finding spreads
An earlier Google DeepMind preprint reported two failure modes in common softmax and cross-entropy implementations:
- Softmax Collapse: When one target logit dominates, its probability can round to exactly one. The corresponding gradient then rounds to zero while smaller gradient components remain, so the components no longer sum to zero as softmax shift invariance requires.
- Softmax Cancellation: When student and teacher logits nearly match, subtracting separately rounded quantities can produce a nonzero gradient. The failure can persist even when the logits are equal.
Amos applies the same numerical scrutiny to divergences and entropy terms used throughout machine learning. These operations often receive nearly identical distributions by design, placing them in the regime where floating-point cancellation is most severe.
Near zero, precision disappears
For categorical distributions, forward KL is commonly evaluated as:
KL(p || q) = Σᵢ pᵢ (log pᵢ - log qᵢ)When p and q nearly match, their log probabilities also nearly match. Subtracting those rounded values discards leading significant digits, while subsequent weighting and summation propagate the error into the divergence and its gradient. Lower-precision formats such as bfloat16 reach this limit sooner, although sufficiently small differences can also expose it in float32.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.