---
title: "NVIDIA's NeMo AutoModel Trains 550B MoE Models 3.7x Faster With One Import"
subtitle: "NeMo AutoModel layers Expert Parallelism and DeepEP on top of Transformers v5, delivering 3.4–3.7x faster MoE fine-tuning with a single import change"
url: "https://alphasignal.ai/news/nvidia-s-nemo-automodel-trains-550b-moe-models-3-7x-faster-with-one-import"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "NVIDIA AI"
company_url: "https://alphasignal.ai/company/nvidiaai"
published: "2026-06-24T16:03:08.000Z"
updated: "2026-06-24T16:50:46.028Z"
type: "News"
topics:
  - "Training Infra"
  - "Llms"
subtopics:
  - "Distributed Training"
  - "Mixture Of Experts"
upvotes: 537
read_time: "5 min"
source_url: "https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel?linkId=100000427835682"
comments: 0
---

# NVIDIA's NeMo AutoModel Trains 550B MoE Models 3.7x Faster With One Import

> NeMo AutoModel layers Expert Parallelism and DeepEP on top of Transformers v5, delivering 3.4–3.7x faster MoE fine-tuning with a single import change

Mixture-of-Experts (MoE) models , architectures where only a small subset of specialized sub-networks, called "experts," activate for each token , have taken over the frontier model landscape. But training them efficiently is a different beast from dense transformers. Routing tokens across hundreds of experts, fusing their computations, and keeping GPUs from stalling on communication overhead requires infrastructure that general-purpose libraries simply weren't built for. NVIDIA's answer is [NeMo AutoModel](https://github.com/NVIDIA-NeMo/Automodel), and it just got a major upgrade by building directly on top of Hugging Face Transformers v5.

## The Problem With Training MoE at Scale

Hugging Face Transformers has become the foundation of the open-source AI ecosystem, and the recent Transformers v5 release strengthened it with first-class support for MoE models. v5 ships the MoE foundations: expert backends, dynamic weight loading, and distributed execution. But v5 still leaves a performance gap on the table. NeMo AutoModel builds on top of v5 by subclassing `AutoModelForCausalLM`, adding Expert Parallelism (EP), DeepEP fused all-to-all dispatch, and TransformerEngine kernels. DeepEP is the piece v5 doesn't have yet: it overlaps communication with expert compute.

Training state-of-the-art MoE models has traditionally required specialists with deep distributed systems knowledge and access to high-end infrastructure. The goal of NeMo AutoModel is to collapse that complexity into a single import swap.

## One Import, 3.7x Faster

The payoff is 3.4–3.7x higher training throughput and 29–32% less GPU memory on fine-tuning MoE models than native Transformers v5, using the same `from_pretrained()` API , a single import line, with no other code changes. The API compatibility is intentional: `NeMoAutoModelForCausalLM` subclasses `AutoModelForCausalLM`, so any existing HF training code works without modification.

![Side-by-side code comparison of NeMo AutoModel vs HuggingFace Transformers API](https://cdn-uploads.huggingface.co/production/uploads/690d0a6c2c5acfe0e1f4777d/VTPq2Wp-RrEcP1eGUJxao.png)

To enable Expert Parallelism across 8 GPUs, you configure a distributed mesh and pass it into `from_pretrained()`:

```
from nemo_automodel import NeMoAutoModelForCausalLM
from nemo_automodel.recipes._dist_utils import create_distributed_setup_from_config
dist_setup = create_distributed_setup_from_config({
    "strategy": "fsdp2",
    "ep_size": 8,
})
model = NeMoAutoModelForCausalLM.from_pretrained(
    "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
    dtype=torch.bfloat16,
    distributed_setup=dist_setup,
)
```

That single call activates FSDP2 sharding, Expert Parallelism, TransformerEngine kernels, and DeepEP dispatch simultaneously.

## Three Engines Under the Hood

The speedup comes from three stacked optimizations, each targeting a different bottleneck in MoE training:

- **Expert Parallelism (EP):** Expert weights are physically sharded across GPUs along the expert dimension. With `ep_size=8` on 8 GPUs, each GPU holds only 1/8 of the expert parameters. For a model like Nemotron-3-Nano-30B-A3B with ~55 GiB of expert weights, this cuts the per-GPU expert footprint from ~55 GiB to ~6.8 GiB.
- **DeepEP fused dispatch:** DeepEP scales token routing to 64+ expert parallelism degrees with highly efficient all-to-all communication and optional permute/unpermute fusion. By leveraging DeepSeek's DeepEP optimization, NeMo AutoModel minimizes communication overhead and maintains balanced expert utilization.
- **TransformerEngine kernels:** NVIDIA Transformer Engine kernels, Megatron-Core DeepEP, and GroupedGEMM minimize communication overhead, increase GPU occupancy, and significantly improve throughput and hardware utilization for large-scale MoE training.

NeMo AutoModel also ships a configurable `BackendConfig` that lets you mix and match kernel backends per component , attention, linear layers, expert matmuls, and the token dispatcher can each be set independently.

## Benchmark Numbers That Matter

NVIDIA tested across two regimes: single-node 30B models and a full fine-tune of a 550B model across 16 nodes. The single-node numbers are the most directly comparable since they pit NeMo AutoModel against the best available Transformers v5 configuration.

![Bar chart showing NeMo AutoModel 3.36x and 3.69x throughput gains and 29-32% memory reduction vs Transformers v5](https://cdn-uploads.huggingface.co/production/uploads/690d0a6c2c5acfe0e1f4777d/rbCVgV6a18c4UcDsiWfZN.png)

On Qwen3-30B-A3B, Transformers v4 doesn't just underperform , it deadlocks entirely. The root cause: v4 stores experts as a `ModuleList` of individual modules, each separately FSDP-wrapped. With different data per GPU rank, different ranks skip different experts, causing mismatched collective operations and an indefinite hang. Transformers v5 fixes this by storing experts as fused 3D parameter tensors.

The 550B result tells a different story. At that scale, Transformers v5 runs out of memory entirely , there's no v5 number to report. NeMo AutoModel is an open library part of the NVIDIA NeMo framework for building custom generative AI models at scale. It builds cleanly on top of v5, adding Expert Parallelism, DeepEP fused all-to-all dispatch, and TransformerEngine kernels. With EP=64 across 128 H100s, the full fine-tune of Nemotron 3 Ultra 550B runs at 815 tokens/sec/GPU and ~293 TFLOP/s/GPU , a regime that simply wasn't accessible before.

## What It Supports (and What It Doesn't)

NeMo AutoModel supports composable parallelism via PyTorch-native FSDP2, HSDP, TP, CP, SP, and PP for distributed training, optimized kernel paths including Transformer Engine, DeepEP, FlexAttn, and Triton, plus MoE routing and DeepEP integration for DeepSeek, Qwen MoE, GPT-OSS, and Nemotron MoE architectures. It also supports FP8 and mixed precision via torchao and Transformer Engine, activation checkpointing, memory-efficient fused linear cross-entropy loss, and sequence packing to reduce padding waste.

The caveat worth noting: for models without hand-tuned NeMo implementations, AutoModel falls back to vanilla HF with lighter optimizations like Liger kernel patching. The full 3.4–3.7x gains apply specifically to the supported MoE architectures (Qwen3, Nemotron, DeepSeek V3, GPT-OSS). The benchmarks also use a balanced routing gate that emulates ideal steady-state expert utilization , real-world numbers with imbalanced token distributions may vary.

## The Bigger Picture

The real unlock here isn't just speed , it's the checkpoint format. Because NeMo AutoModel rides v5's reversible weight conversion to load each model, `save_pretrained()` still emits standard HF-format checkpoints that tools like vLLM and SGLang can load. Train fast with NeMo, deploy anywhere in the HF ecosystem. That's a meaningful workflow improvement over frameworks that require proprietary checkpoint formats.

NeMo AutoModel is open-source and available now on [GitHub](https://github.com/NVIDIA-NeMo/Automodel). Installation and recipes are documented in the [official docs](https://docs.nvidia.com/nemo/automodel/latest/get-started/installation), and the [HuggingFace integration guide](https://huggingface.co/docs/transformers/community_integrations/nemo_automodel_finetuning) covers the fine-tuning workflow end to end. It's free and open-source.

## Sources

- [X post by @nvidiaai](https://x.com/nvidiaai/status/2069813582825418828)
- [Accelerating Transformers Fine-Tuning with NVIDIA NeMo AutoModel](https://huggingface.co/blog/nvidia/accelerating-fine-tuning-nvidia-nemo-automodel)
- [NeMo AutoModel GitHub Repository](https://github.com/NVIDIA-NeMo/Automodel)
- [Democratizing Large-Scale MoE Training in PyTorch — NVIDIA Technical Blog](https://developer.nvidia.com/blog/accelerating-large-scale-mixture-of-experts-training-in-pytorch/)
- [NeMo AutoModel HuggingFace Integration Docs](https://huggingface.co/docs/transformers/community_integrations/nemo_automodel_finetuning)