Skip to main content
Back

Thinking Machines' Inkling 2 Mini Loops 24 Layers Into 48 Applications

vLLM ships day-0 kernels for Thinking Machines' Inkling 2 Mini, a 26B looped transformer that runs its middle 12 layers three times per token.

  • vLLM shipped day-0 support for Inkling 2 Mini, a 26B looped MoE from Thinking Machines.
  • Middle 12 of 24 layers run three times with shared weights, giving 48 layer visits per token.
  • Cross-pass messages use a gated rank-64 controller; vLLM fuses all four reads per layer into one op.
  • 26.4B total, 3.4B active, 5.4B effective active after loops; 1M context, Apache 2.0.
  • Three passes cost about 2.5x decode latency and halve KV cache capacity versus single-pass.
  • Variable-loop research checkpoint lets you trade quality for speed at inference time.

Thinking Machines has released Inkling 2 Mini, an Apache 2.0 open-weight mixture-of-experts model with 26.4 billion parameters. Its middle 12 layers reuse the same weights across three passes, increasing computation without adding more stored layers. vLLM added launch-day support, including fused kernels for the messages exchanged between passes.

Twenty-four layers, 48 applications

Component Configuration
Architecture Decoder-only transformer with sparse MoE feed-forward layers
Stored decoder layers 24
Layer applications 48 per token
Loop core Layers 6–17, executed three times
Expert routing 8 of 128 experts selected per token
Attention Hybrid local and global layers
Parameters 26.4B total, 3.4B active, 5.4B effective active

The 3.4B active figure counts the unique weights selected along a token’s route. Each additional trip through the shared core adds about 1.0B parameter-equivalents of computation, producing 5.4B effective active parameters after three passes while retaining 26.4B stored parameters.

Messages keep the loop connected

The decoder divides its layers into a six-layer prelude, a 12-layer loop core, and a six-layer coda. Every core pass begins with the prelude’s output, and the coda consumes the final core output after the third pass.

Beginning with the second pass, the query, key, value, and MLP inputs in each core layer receive information from the preceding pass. Each message comes from the prior pass’s two attention-residual blocks and passes through a gated rank-64 controller, a low-rank projection that limits the added computation. The gate initially suppresses cross-pass messages, allowing training to increase their influence gradually.

Repeated execution also increases inference state. Each layer application needs its own key-value cache and convolution-state cache, producing 48 sets of each across the unrolled decoder. vLLM registers them as ordinary cache layers, preserving prefix caching and CUDA graph capture.

vLLM fuses the loop

The vLLM implementation combines all four message reads in a layer into one attention-residual operation. It also stacks the four controller weight sets so GPUs can execute their projections as batched general matrix multiplications, or GEMMs.

The integration supports the bundled DSpark speculator for speculative decoding. DSpark proposes several likely tokens, and the main model verifies them together, reducing the number of full decode steps when those proposals are accepted.

Launch it with DSpark

vllm serve thinkingmachines/Inkling-2-Mini \
  --tensor-parallel-size 1 \
  --reasoning-parser inkling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
  • --tensor-parallel-size 1 keeps the model on one accelerator.
  • --reasoning-parser inkling selects Inkling’s parser for reasoning content in API responses.
  • num_speculative_tokens allows DSpark to propose up to seven tokens per speculative step.
  • draft_sample_method uses probabilistic sampling for those proposals.

According to the serving recipe, the configuration targets about 4,400 aggregate output tokens per second per GPU while sustaining 100 output tokens per second per user on 32K-context agentic sessions with a single GB200. Results will vary with batch size, prompt length, output length, and acceptance rate. The model currently requires a nightly vLLM build or the vllm/vllm-openai:inkling-2-mini image until support reaches a stable release.

Deployment memory depends on all 26.4B stored parameters plus runtime caches. At 16-bit precision, the weights alone require roughly 53GB before cache and framework overhead, so the 3.4B active-parameter figure should be treated as a compute measure.

Three passes raise latency and reduce cache capacity

vLLM’s single-GB200 measurements show that the three-pass configuration takes about 2.5 times as long per decode step and supports roughly half as many cached tokens as the single-pass configuration.

Mode Decode step Relative time KV-cache capacity
One pass 1.51 ms 1.00× 352K tokens
Three passes 3.85 ms 2.55× 175K tokens

Thinking Machines reports that a non-looping comparison model used 7.5% less compute during training. Matching the looped model’s loss on 512K-token agentic traces required 140% more inference compute, suggesting that shared depth can shift computation from training into repeated inference steps more efficiently for that workload.

Thinking Machines also trained a separate variable-loop research model that accepts one, two, or three passes. One- and two-pass settings reduce inference time, with slightly lower generation quality reported. This speed-quality control is specific to the variable-loop variant; fixed-pass checkpoints should use the pass count applied during training.

Why share depth?

Looped transformer designs have appeared in prior research, but open checkpoints with optimized serving support remain uncommon. Inkling 2 Mini executes 48 sequential layer applications, matching Inkling 2 Small’s depth while using about one twenty-second of its total parameters, according to Thinking Machines.

The architecture tests the hypothesis that middle layers perform more reusable representation processing, while early and late layers handle more of the translation between token embeddings and internal states. Repeating the middle blocks concentrates additional computation there without storing another set of weights.

More hidden work, thinner traces

Repeated hidden-state computation changes what external monitoring can observe. Chain-of-thought logs contain emitted tokens, while the extra loop passes occur inside hidden activations. Systems that evaluate agents through visible reasoning traces therefore have less direct evidence of the computation performed between tokens, a behavior Thinking Machines examined with the variable-loop model.

The Apache 2.0 checkpoint and vLLM integration give developers a direct way to compare shared-depth MoE models with dense and conventionally layered alternatives. The practical evaluation points are measurable: weight memory, per-pass latency, cache capacity, speculative-decoding acceptance, and performance on long-context agentic workloads.