White Circle's Halo Open-Source Framework Trains 20B Models 2.8x Faster
White Circle open-sourced Halo, a distributed training framework that keeps models in native HuggingFace format while hitting up to 2.8x TRL throughput.
- White Circle open-sourced Halo, a distributed training framework for LLMs and multimodal models.
- Delivers 2.3–2.8x stock TRL throughput on gpt-oss-20b with lower peak memory.
- Models stay in native HuggingFace format; new families need only ~100 LOC wrapper.
- Supports EP, CP, TP, ETP parallelism, LoRA/QLoRA, and async multi-turn RL via SGLang.
- Custom bf16 AdamW halves optimizer memory; fits a 20B model on one GPU.
- GLM-4.7-Flash-Coder trained with Halo lifted SWE-rebench-V2 from 33% to 42%.
Halo brings distributed training to stock Hugging Face models
White Circle has open-sourced Halo, the training framework it uses for its released models. Halo targets teams whose models have outgrown standard Hugging Face TRL workflows but do not justify the engineering cost of porting architectures into Megatron-LM or NeMo. It adds distributed training and reinforcement learning while preserving Hugging Face model classes and checkpoints.
Halo accepts an unmodified Hugging Face model and produces a checkpoint loadable through the standard from_pretrained interface. Teams can keep existing architectures, avoid conversion scripts, and add a model family with roughly 100 lines of wrapper code, according to White Circle.
A controlled throughput comparison
White Circle reports 2.3 to 2.8 times the throughput of stock TRL when training OpenAI’s gpt-oss-20b, along with lower peak memory use. Both systems use FlashAttention-4, Liger kernels with fused linear cross-entropy, and grouped-GEMM expert kernels. The comparison therefore isolates differences in the trainer, optimizer, and parameter-sharding strategy.
The project’s benchmark report provides these results:
| Configuration | Reported result |
|---|---|
gpt-oss-20b, 4K tokens, batch 4, EP1, gradient checkpointing disabled |
Peak throughput of 24,456 tokens per second per GPU |
gpt-oss-20b, 4K tokens, batch 1, EP8 |
26 GB per GPU, compared with 47.6 GB for TRL with ZeRO-3 |
gpt-oss-20b, 64K context |
About 2.1 times TRL throughput |
gpt-oss-20b, 256K context |
About 1.3 times TRL throughput; context parallelism halves the per-device memory footprint |
Qwen3.5-4B, 16K context |
Halo leads on a dense vision-language model without experts to distribute |
These are project-reported benchmarks, and distributed-training results depend heavily on GPU topology, software versions, sequence length, and batch shape. The repository and report provide the configurations needed to inspect or reproduce the comparisons.
Four ways to divide the work
Halo combines four parallelism strategies so teams can distribute model weights, experts, and long sequences according to the workload:
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
