Skip to main content
Back

TIPOv2 Rebuilds Itself as a Sparse AI to Write Your Image Prompts

TIPOv2 rebuilds the popular prompt-expansion model as a 991M sparse MoE with 193M active parameters, five datasets, and a new training stack.

  • TIPOv2-1B-A200M is a sparse MoE prompt-expander that turns short tags into detailed text-to-image prompts
  • 991M total parameters with only 193M active per token via 64 experts, top-8 routing
  • Trained on refreshed Danbooru plus Nozomi, CC12M, CoyoHD-11M, and LAION-COCO-13M
  • Uses MXFP8 kernels, Muon optimizer, and aux-loss-free load balancing on 4x RTX 5090s
  • Ships as fp16 safetensors and GGUF, plug-in via z-tipo-extension for WebUI/ComfyUI
  • Released under Kohaku License 1.0; based on TIPO arXiv paper 2411.08127

If you have spent any time coaxing Stable Diffusion or an anime-style diffusion checkpoint into producing something coherent, you have probably wished the model would just fill in the twenty booru tags you did not think to write. That is what TIPOv2-1B-A200M does, and the second generation is a full rebuild rather than a fine-tune.

TIPO improves text-to-image generation by text presampling: a small language model expands a short user prompt into a detailed one before the diffusion model ever sees it, narrowing the range implied by a brief prompt to a specific description that still matches the original. Instead of asking the user to write 200 tokens of booru tags and natural language, TIPO samples that expansion from a distribution learned over real caption data.

Sparse where v1 was dense

The headline change is architectural. v1 was a 500M dense LLaMA-style decoder. v2 is a sparse mixture-of-experts model built on the KohakUwU MoE architecture, a DeepSeekMoE-style design from the author's KohakUwULLM training project.

The numbers are worth pausing on:

  • 990.8M total parameters, with only 193.1M active per token
  • 64 routed experts with top-8 selection, plus 1 shared expert that is always on
  • 16 layers (layer 0 dense, the remaining 15 MoE), hidden size 768
  • GQA attention with 12 heads and 2 KV heads, plus QK-norm
  • RoPE with 4096 context, up from v1's 1024
  • Aux-loss-free load balancing via a sigmoid-scored router with a selection-only bias