Humans& Ships Weight Teleportation to Sync 445 GB AI Models in 220 ms

Humans& has published Weight Teleportation, an open-source pipeline for moving updated model weights from reinforcement-learning trainers to rollout samplers. In its reference benchmark, the system compressed a 445 GB model update to 2.65 GB, completed end-to-end synchronization in under four seconds, and paused sampling for 219.6 ms.

The design combines bitwise deltas, nested lossless compression, CPU staging, Blackwell’s fixed-function Decompression Engine, and Triton kernels. Its measured application speed reaches the decompression hardware’s throughput limit.

Why weight movement stalls rollout

Large-scale RL systems use trainers to update a policy and samplers to run that policy while generating trajectories. Those samplers need regular weight updates because growing version lag changes the policy that produced the training data, while blocking updates leave inference GPUs idle.

Full checkpoints can span hundreds of gigabytes. Sending them frequently requires substantial network capacity, and loading them can evict the key-value cache used for inference. Weight Teleportation reduces the transferred bytes and performs most receiver work while rollout decoding continues.

The benchmark behind the claim

The reference workload uses an NVFP4 GLM-5.3 checkpoint containing 445 GB of canonical weight tensors. The trainer side spans eight nodes with 64 B300 GPUs, and each simulated step mutates 0.085% of the checkpoint as a proxy for a steady-state RL update.

• Canonical weight size: 445 GB
• Trainer hardware: 64 B300 GPUs across eight nodes
• Mutation rate: 0.085% per step
• Compressed payload: 2.65 GB, about 0.60% of the checkpoint
• End-to-end synchronization: under four seconds
• Sampler pause: 219.6 ms

The mutation rate is central to the result. Denser or less compressible updates would increase payload size, network time, staging requirements, and potentially the sampler pause.

Bitwise deltas expose the zeros

The trainer treats the old and new weights as flat bit buffers and computes their exclusive OR, or XOR. Each zero marks an unchanged bit, while each one marks a flipped bit. The receiver applies the same XOR operation to its old weights and reconstructs the new weights exactly.

1. The trainer XORs the previous and current weight buffers.
2. Lossless codecs compress the resulting delta.
3. The receiver decompresses the delta into temporary layer buffers.
4. A Triton kernel XORs each buffer into the live weights.

A raw XOR mask occupies as many bytes as the original weights. Its compression ratio comes from repeated zeros when relatively few bits change. Because reconstruction uses bitwise operations, the method preserves every bit and works across floating-point and low-precision formats such as NVFP4.

Both endpoints must share the same base weights, byte layout, and version. A missing or misidentified update would produce the wrong parameters, so production integrations need strict checkpoint and delta version tracking.

Two codecs produce the useful stream

A direct GPU codec leaves too many bytes on the wire in this workload. GPU LZ4 produced a 5.36 GB payload, 6.1 times the 0.876 GB generated by a CPU Zstd baseline. Fully decoding the tensor delta on the CPU would sacrifice the direct hardware path into GPU memory, so the pipeline nests two codecs.

Snappy creates the larger first-stage payload, but its simpler copy commands give Zstd more repetitive structure to compress. Zstd reduces the Snappy stream to about 13% of its original size, compared with roughly 75% for the LZ4 stream.

Blackwell turns decoding into a stream

The receiver decodes the outer Zstd layer on the CPU while rollout inference continues. It places the resulting Snappy stream in pinned host memory, then uses Blackwell’s Decompression Engine to read that stream directly and write decompressed layer data into high-bandwidth GPU memory.

CUDA exposes this path through the driver allocation flag CU_MEM_CREATE_USAGE_HW_DECOMPRESS. The Decompression Engine is a fixed-function block separate from the GPU’s streaming multiprocessors, allowing hardware decompression to overlap with other compute.

A double-buffered pipeline keeps both engines occupied. The Decompression Engine writes one layer into buffer A while a Triton kernel applies the delta from buffer B to the live weights, after which the buffers swap roles. Only two layers need temporary GPU storage, so the implementation can use memory already reserved for inference activations and preserve the key-value cache.

The Triton XOR work fits behind decompression latency in the reported workload. Once that overlap is complete, the Decompression Engine sets the lower bound for the blocking application phase.

Measured costs

Most of the 2.355 seconds of receiver work runs without blocking rollout because the outer decode uses CPU resources and pinned host memory. The sampler pauses during the final hardware-decompression and XOR streaming phase.

One delta system covers several workflows

Version-specific deltas let the same infrastructure handle synchronization, recovery, evaluation, and deployment without separate checkpoint-transfer mechanisms.

• Evaluation workers: An evaluator several steps behind receives a delta computed against its last loaded version.
• Fault recovery: A restarted rollout or evaluation worker loads its base checkpoint and requests a delta from that exact version.
• RL startup: A run resumed from a trained checkpoint uses LZ4 plus Zstd for the larger accumulated delta. The smaller inner LZ4 stream limits receiver staging memory.
• Remote deployment: A serving process starts from a shared base checkpoint and requests current weights through /update_weights_from_gpu_delta.

The authors also present full-parameter deltas as an option for serving workloads that might otherwise use LoRA adapters. The delta format can represent arbitrary weight changes without imposing a rank limit, while its size depends on bit-level compressibility and access to a shared base checkpoint.

Code lands in Miles and SGLang

Miles contains orchestration, distributed GPU compression, fault tolerance, and delta checkpointing. SGLang contains the heterogeneous receiver pipeline, Decompression Engine overlap, Triton kernels, and one-shot weight-update API.

• SGLang PR
• Miles PR
• nvCOMP docs

The accelerated receiver path targets Blackwell hardware, including B200 and B300 systems, and depends on CUDA driver support for hardware-decompression allocations. Teams using earlier GPU generations would need another decode path and should expect different pause times.

Where the benchmark may bend

• Update density: The reported workload mutates 0.085% of the checkpoint. Compression will weaken as more bits change.
• Shared state: Sender and receiver need identical base weights, tensor layouts, and version metadata.
• Host resources: The receiver needs pinned CPU memory for the inner compressed stream and enough CPU throughput for Zstd.
• Hardware: The shortest blocking path relies on Blackwell’s Decompression Engine and its supported codecs.
• Network: End-to-end time still depends on bandwidth, latency, topology, and concurrent traffic.
• Validation: The published figures come from the project team’s proxy workload and await independent reproduction on complete RL training runs.

Humans& reports that the compressed pipeline can outperform intra-cluster peer-to-peer RDMA for many practical workloads because it sends fewer bytes and overlaps receiver work. Cross-datacenter sampling becomes feasible when the compressed payload, available bandwidth, and permitted policy lag fit the training loop’s requirements.

The benchmark shows how a sparse, exact weight delta can become a hardware-decodable stream whose final application takes about 220 ms. For Blackwell-based RL systems, the available code provides a concrete integration path and a measurable baseline for evaluating update density, network cost, host-memory pressure, and sampler downtime.