Tencent Cuts AI Training Time by 29% With a Simple Batch Size Trick

Tencent Hunyuan researchers report up to a 29% reduction in time to target for tested Group Relative Policy Optimization runs after increasing batch size and retuning Adam’s learning rate. Their study identifies when larger batches accelerate online reinforcement learning for large language models and when they consume additional compute without improving completion time.

Online RL generates model responses, scores them, computes gradients, updates the model, and repeats the cycle. Response generation often dominates runtime, so batch size affects both optimization efficiency and the rate at which GPUs produce training data.

One batch knob, two bottlenecks

The study tests Group Relative Policy Optimization (GRPO), which compares rewards within groups of responses, and Proximal Policy Optimization (PPO), which trains an actor policy alongside a critic that estimates value. Both methods must balance samples-to-target against end-to-end throughput.

For a fixed performance target, approximate training time depends on how many samples the model needs and how quickly the system processes them. A larger batch pays off when its throughput gain exceeds any increase in samples-to-target.

The paper also distinguishes the training batch from generation concurrency. The training batch controls how many examples contribute to an optimizer step. Generation concurrency controls how many sequences the decoding engine processes together. Teams can tune these values separately, subject to memory and pipeline constraints.

The square root keeps runs aligned

The researchers preserve sample efficiency by scaling Adam’s learning rate with the square root of the batch-size ratio:

eta(B) = eta_0 * sqrt(B / B_0)

Here, B_0 and eta_0 are the reference batch size and learning rate. Doubling the batch multiplies the learning rate by sqrt(2). Increasing it eightfold multiplies the rate by sqrt(8).

Under this rule, learning curves from different batch sizes largely overlap when plotted against cumulative samples. The paper calls this behavior batch-size invariance. A batch of 1,024 can then reach the same score after roughly the same number of responses as a batch of 128 while requiring eight times fewer optimizer steps.

The observed invariance ended at algorithm-specific critical batches:

Crossing those boundaries caused the sample-indexed curves to diverge even after learning-rate scaling. The study establishes the boundary empirically, so each model, algorithm, and workload still requires validation.

Decoding supplies the wall-clock gain

Autoregressive decoding at low concurrency is often limited by memory bandwidth. GPUs repeatedly load model weights while generating one token at a time. Larger active batches spread that weight traffic across more sequences, improving hardware utilization until compute or KV-cache capacity becomes the constraint.

Actor updates scale differently because training kernels are typically compute-bound. This limits the end-to-end benefit once additional update time outweighs faster rollout generation.

PPO also showed an actor-critic asymmetry. Actor gradient noise declined steadily as batches grew, while critic gradients remained noisier. Separate batch sizes or update schedules may therefore improve PPO efficiency, although the paper does not fully characterize the critic’s optimal batch range.

An unchanged rate consumes the gain

The GRPO ablation shows the cost of increasing batch size without retuning the learning rate. Doubling the batch raised samples-to-target by 67%. At a prompt batch of 256, denoted P=256, the fixed-rate control took 1.42 times as long as the reference run. Its 1.17-fold throughput improvement could not offset the 1.67-fold sample penalty.

Profile in two passes

A production evaluation can separate optimization alignment from systems tuning:

1. Establish a baseline. Record the reference batch B_0, learning rate eta_0, samples-to-target, rollout throughput, update time, and total wall-clock time.
2. Scale the optimizer. For each candidate batch B, initialize Adam with eta_0 * sqrt(B / B_0).
3. Check sample efficiency. Plot evaluation performance against cumulative generated responses and reject batch sizes whose curves diverge materially from the baseline.
4. Raise decode concurrency. Increase the active generation batch until throughput plateaus, latency becomes unacceptable, or KV-cache memory is exhausted.
5. Measure the full loop. Include rollout generation, reward scoring, actor updates, critic updates, synchronization, and idle time.
6. Select the fastest valid run. Choose the configuration with the lowest measured time to the target score, then confirm it across repeated runs.

Where the evidence stops

• Model coverage: The experiments use Qwen3-30B-A3B and an internal Hunyuan mixture-of-experts model with 3B active parameters.
• Task coverage: The workloads cover math, science, and logic rather than a broad set of RL tasks.
• Finite scaling range: Batch-size invariance fails beyond a model- and algorithm-dependent threshold.
• Critic tuning: The PPO critic may have a different critical batch size, which remains incompletely characterized.
• Deployment dependence: The best generation batch varies with hardware, decoding engine, response-length distribution, precision, and available KV-cache memory.

Teams can use square-root learning-rate scaling as a starting point, verify sample-indexed alignment, and then profile generation concurrency. That sequence turns batch sizing into a measurable optimization problem centered on end-to-end time to target.