DeepSeek Opens DeepInfer, Exposing the Engine Running V4.1-Flash

DeepSeek has released DeepInfer, the inference framework behind DeepSeek-V4.1-Flash, and deepseek-recipe, its request-rendering and response-parsing layer. The repositories expose two previously private parts of the hosted API: model execution and protocol semantics.

The engine behind V4.1-Flash

DeepInfer packages the core execution features DeepSeek uses for large-scale serving. Its architecture builds on vLLM and incorporates ideas from SGLang and TensorRT-LLM, giving experienced inference engineers a familiar base with DeepSeek-specific implementations layered above it.

The initial release supports DeepSeek-V4.1-Flash and exposes offline inference. It does not provide a complete managed endpoint, so production deployments still need service infrastructure around the engine.

Kernels, routing, and faster decoding

Batch-invariant inference and deterministic sampling currently require NVIDIA hardware. These controls help teams compare deployments, reproduce failures, and test changes without batch composition altering the output.

A deliberately narrow hardware target

DeepInfer currently supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, with separate dependency files for each platform. The repository reports benchmark configurations using four SM100 GPUs for prefill and 64 for decoding, all within one scale-up domain.

Prefill processes the input prompt and populates the key-value cache. Decode generates subsequent tokens, usually one step at a time. Separating these phases allows operators to assign different hardware pools and scaling policies to their distinct workloads.

The service layer remains yours

Production deployments need several components beyond DeepInfer’s offline engine:

• KV-cache management: storage and coordination for the attention state reused during token generation.
• Execution disaggregation: separate workers for the phases the repository describes as VE, prefill, and decode. DeepSeek says most required engine functionality is already implemented.
• Replica load balancing: request distribution across data-parallel prefill and decode workers.
• API transport: HTTP handling, authentication, quotas, retries, observability, and admission control.

These boundaries make DeepInfer most useful to teams prepared to operate a distributed inference service. The repository supplies execution primitives and optimization examples; operators remain responsible for reliability and capacity management.

API semantics become reference code

The deepseek-recipe project standardizes how applications translate API payloads into model input and turn raw generations back into API responses. Its Rust libraries and Python bindings convert supported request formats into an internal Conversation representation, render that conversation as a DeepSeek prompt, and encode it with an attached tokenizer.

After inference, the library parses reasoning content and tool calls, then produces streaming or complete responses in the corresponding API format. Supported inputs and behaviors include:

• Messages, Chat Completions, and Responses requests
• Text and image content
• Thinking mode and reasoning extraction
• Client-defined tool calls
• Chat Completions prefix continuation
• Prompt rendering for DeepSeek V4 and V4.1

The library also applies mode-specific context defaults:

A minimal Python integration

The Python package installs with pip install deepseek-recipe. A minimal request conversion and prompt-rendering flow looks like this:

from deepseek_recipe import (
    ChatCompletionRequest,
    ConversionOptions,
    DeepseekV41Encoding,
)

request = ChatCompletionRequest({
    "model": "DeepSeek-V4.1-Flash",
    "messages": [
        {"role": "user", "content": "Hello"},
    ],
})

converted = request.convert(ConversionOptions())
rendered = DeepseekV41Encoding().render_conversation(
    converted.conversation
)

print(rendered.prompt)

The application supplies model inference, HTTP transport, and tool execution. DeepInfer can perform the model’s forward pass, while deepseek-recipe handles the request and response conventions around it.

How the pieces connect

1. The application receives a supported API request.
2. deepseek-recipe converts the payload into a conversation, renders the prompt, and produces token IDs.
3. DeepInfer runs DeepSeek-V4.1-Flash and generates output tokens.
4. deepseek-recipe parses text, reasoning content, and tool calls into the requested response format.
5. The application streams or returns the result and executes any requested tools.

What changes for self-hosters

Community implementations have had to reconstruct DeepSeek’s prompt templates, tool-call encoding, reasoning boundaries, and response parsing from observed API behavior. deepseek-recipe now provides an inspectable implementation for those semantics, reducing compatibility errors between hosted and self-managed deployments.

DeepInfer gives serving engineers concrete implementations of the techniques DeepSeek uses to reduce inference work and coordinate large mixture-of-experts deployments. Its source tree covers speculative decoding, sliding-window prefill reduction, deterministic execution, asynchronous scheduling, and expert load balancing on top of vLLM-derived primitives.

Immediate adoption fits deployments using DeepSeek-V4.1-Flash on supported hardware with an existing service platform. deepseek-recipe has broader utility for gateways and compatibility layers that need DeepSeek’s request, prompt, and response behavior without rebuilding those rules independently.