Hugging Face Open-Sources Nanowhale, a Tiny DeepSeek-V4 That Fits on One GPU
Hugging Face open-sourced nanowhale, a 110M-parameter DeepSeek-V4 clone that packs MLA, MoE, and MTP into a single H100-trainable repo.
PRO- Hugging Face open-sourced nanowhale, a 110M-parameter DeepSeek-V4 clone trained from scratch.
- Implements MLA, MoE (4 routed + 1 shared expert, top-2), Hyper-Connections, and Multi-Token Prediction.
- The pretraining and SFT pipeline was executed end-to-end by the ml-intern agent.
- Base and instruct checkpoints available: nanowhale-100m-base and nanowhale-100m.
- Trained on 1x H100 80GB in bf16, 2.6B tokens, 72ms/step with torch.compile; MIT licensed.
- Known issues: bf16 overflow forces fp32 inference, and 129K vocab eats 37% of parameters.
Hugging Face has open-sourced nanowhale, a compact reference implementation of the DeepSeek-V4 architecture that fits on a single GPU. The ~110M parameter language model is trained from scratch, and the repo ships all the code, configs, and tokenizer used to pretrain and fine-tune it. Think of it as nanochat for the DeepSeek family: a minimal, hackable codebase where you can actually see how the moving parts fit together.
There's a twist in how it got built. The model was fully pretrained by an agent inspired by Karpathy's nanochat. The Hugging Face team gave ml-intern the task of training a tiny MoE with all the architectural advancements of DeepSeek v4, and it trained a 100M-parameter MoE end-to-end. Regardless of who wrote the code, what's shipping today is a clean, MIT-licensed repo you can clone and run.
A miniature DeepSeek-V4, feature complete
Nanowhale isn't trying to compete on quality with frontier models. It gives you a readable end-to-end implementation of every architectural trick DeepSeek uses at frontier scale, shrunk to something you can inspect and modify. The model includes Multi-Head Latent Attention (MLA) with 8 heads and 1 KV head (MQA), head_dim=96 split into 32 RoPE + 64 NoPE, q_lora_rank=160; Mixture-of-Experts with 4 routed + 1 shared expert, top-2 routing, SwiGLU FFN; Hyper-Connections with hc_mult=4 and Sinkhorn routing over 2 iterations; and Multi-Token Prediction with 1 next-token prediction layer.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
