DeepSeek Releases DeepSeek-LLM-7B-Chat, a Bilingual Open-Source Model Running on One GPU

DeepSeek-LLM-7B-Chat is a 7-billion-parameter instruction-tuned language model from DeepSeek AI, trained entirely from scratch on 2 trillion tokens of English and Chinese text. It sits in a broader model family that also includes a 67B variant, and it is fully open-source with a commercial-use license. With nearly 50,000 downloads and over 3,000 likes on Hugging Face, it has quietly become one of the more adopted small open-source chat models available today.

Built from scratch, not borrowed

DeepSeek-LLM-7B-Chat is initialized from deepseek-llm-7b-base and fine-tuned on extra instruction data. That base model was not a fork of LLaMA or another existing checkpoint. It was trained from scratch on a vast dataset of 2 trillion tokens in both English and Chinese, and released open-source to foster research.

The architecture follows a decoder-only transformer design, but with some deliberate tweaks. Both the 7B and 67B versions use a pre-norm design with the SwiGLU activation, which stabilized training. DeepSeek LLM 7B is a 30-layer network, while DeepSeek LLM 67B has 95 layers. The team also made an unconventional choice on the learning rate schedule: the training process used a multi-step learning rate scheduler instead of the typical cosine scheduler, which maintained performance while enabling continual training by allowing researchers to reuse earlier training phases when scaling up.

The scaling law angle

What makes this release interesting beyond the model itself is the research philosophy behind it. The scaling laws described in previous literature present varying conclusions, which casts a dark cloud over scaling LLMs. DeepSeek delved into the study of scaling laws and presented distinctive findings that facilitate the scaling of large-scale models in two commonly used open-source configurations, 7B and 67B.