Hidden Math Bug Silently Corrupts AI Training Gradients Near Equal Distributions

A new arXiv preprint extends recent softmax findings to KL and JSD divergences, showing standard code silently corrupts gradients when distributions are close.

·
·
·
Hidden Math Bug Silently Corrupts AI Training Gradients Near Equal DistributionsPRO
  • New arXiv preprint by Brandon Amos extends softmax numerical-bug findings to KL, JSD, and entropy.
  • Standard divergence code loses precision via cancellation when two distributions are nearly equal.
  • Affects knowledge distillation, KL-regularized RL (PPO, GRPO), and proximal optimization methods.
  • Also covers Monte Carlo KL estimators k1, k2, k3 used in policy gradient training.
  • Proposes stable reformulations with example implementations, validated via property-based tests.
  • Builds on Barron et al. (2026) softmax and cross-entropy fix from Google DeepMind.

Brandon Amos’s short arXiv preprint extends recent work on numerical failures in softmax and cross-entropy. It finds related floating-point problems in forward and reverse Kullback-Leibler divergence, generalized Jensen-Shannon divergence, Monte Carlo KL estimators, and categorical entropy. The paper also provides stable reformulations and example code.

A softmax finding spreads

An earlier Google DeepMind preprint reported two failure modes in common softmax and cross-entropy implementations:

  • Softmax Collapse: When one target logit dominates, its probability can round to exactly one. The corresponding gradient then rounds to zero while smaller gradient components remain, so the components no longer sum to zero as softmax shift invariance requires.
  • Softmax Cancellation: When student and teacher logits nearly match, subtracting separately rounded quantities can produce a nonzero gradient. The failure can persist even when the logits are equal.

Amos applies the same numerical scrutiny to divergences and entropy terms used throughout machine learning. These operations often receive nearly identical distributions by design, placing them in the regime where floating-point cancellation is most severe.

Near zero, precision disappears

For categorical distributions, forward KL is commonly evaluated as:

code
KL(p || q) = Σᵢ pᵢ (log pᵢ - log qᵢ)

When p and q nearly match, their log probabilities also nearly match. Subtracting those rounded values discards leading significant digits, while subsequent weighting and summation propagate the error into the divergence and its gradient. Lower-precision formats such as bfloat16 reach this limit sooner, although sufficiently small differences can also expose it in float32.

Pro article

This story is for Pro members

You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.

Trending
  • No trending articles

Comments

avatar

Next Reads