Soup Trains Llama-3.1-8B on a 4 GB Laptop GPU for Free
Layer streaming plus 4-bit quantization pins an 8B model into 3.32 GB of VRAM, turning laptop GPUs into real fine-tuning boxes.
PRO- Soup fine-tunes an 8B LLM on a 4 GB laptop GPU at 119.6 tok/s, 3.32 GB peak.
- Layer streaming keeps the frozen base in RAM, sends one decoder layer at a time to VRAM.
- Bit-exact against resident training, replicated on H100 at 113 tok/s same 3.32 GB.
- DPO, ORPO, SimPO, KTO stream too, with the reference model costing no extra weights.
soup shipnow includes a noise-floor and benign-prompt detector to catch gate gaming.- Install:
pip install "soup-cli[train]", Apache-2.0, Python 3.10 to 3.12, Zenodo paper.
Fine-tuning a text model that would not otherwise fit into your GPU used to mean paying for a rented H100 or wrestling with sharding across multiple cards. A new open-source CLI called Soup collapses that workflow into a single YAML file, and its headline trick is a technique the author calls layer streaming, which trains an 8B model on a 4 GB laptop GPU.
The pitch is unusually concrete: Llama-3.1-8B-Instruct plus NF4 quantization runs at 119.6 tokens per second with a 3.32 GB peak on an RTX 3050 Laptop 4 GB, bit-exact against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB. Everything is Apache-2.0, offline, and driven from one command.
Streaming layers instead of loading them
The idea is simple to state. The frozen base model stays in host RAM, and a dedicated CUDA stream feeds it to the GPU one decoder layer at a time, so peak VRAM is bounded by a single layer rather than the whole model. Only the LoRA adapter, the current layer, and a small double-buffer live on the card.
Combine that with 4-bit NF4 quantization of the streamed base, which shrinks the store roughly fourfold, and an 8B model fits inside 3.32 GB with room to spare. In the config, it is a single opt-in key:
training:
stream_layers: true # base streams out of VRAM; only the adapter trains
quantization: 4bit # NF4 store, ~4x smaller
batch_size: 4 # bigger batches amortise the weight read
stream_source: auto # RAM when it fits, NVMe when it does notThis story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
