Skip to main content
Back

Xiaohongshu's dots.ocr Beats Gemini 2.5 Pro on Multilingual Parsing at 3B Parameters

dots.ocr / dots.mocr is a 3B open-source VLM that parses documents, tables, formulas, and 126 languages in a single model pass, beating Gemini 2.5 Pro on multilingual benchmarks.

Pro

  • Single model, full pipeline: dots.ocr / dots.mocr replaces multi-stage OCR pipelines with one 3B VLM doing layout detection, text recognition, and reading-order inference jointly.
  • Multilingual SOTA: On the XDocParse benchmark (126 languages), it beats Gemini 2.5 Pro by 29.5% on overall edit distance and halves its text error rate.
  • Evolved into dots.mocr: The successor adds SVG chart parsing, web screenshot parsing, and scene text spotting, with a dedicated dots.mocr-svg variant for structured graphics.
  • Scalable data engine: Training used a 3-stage pipeline with Qwen2.5-VL-72B as teacher, a 7B student auto-labeler, and a human-in-the-loop correction loop producing 15k+ high-signal samples.
  • vLLM-native deployment: Officially integrated into vLLM v0.11.0; parse PDFs and images with a single Python command, outputting JSON, Markdown, and annotated images.
  • Open-source MIT license: Weights on HuggingFace, live demo at dotsocr.xiaohongshu.com, 8.9k GitHub stars.

Document parsing has always been a pipeline problem: one model detects layout boxes, another runs OCR, a third handles tables, and a fourth tries to stitch the reading order back together. Every handoff is a chance for errors to compound. dots.ocr, from Xiaohongshu's (Rednote) hi lab, collapses that entire stack into a single 3B-parameter vision-language model that handles layout detection, text recognition, and reading-order inference in one forward pass.

The project has since evolved into its successor, dots.mocr, which extends the original with SVG-based structured graphics parsing, web screenshot parsing, and scene text spotting. Both models are open-source under the MIT license, with weights on HuggingFace and a live demo available to try right now.

The pipeline problem it solves

Traditional document parsing tools like MinerU, Marker, and Docling chain together specialized models: a layout detector (often a YOLO variant), an OCR engine, a formula recognizer, and a table parser. By leveraging a single vision-language model, dots.ocr offers a significantly more streamlined architecture than conventional methods that rely on complex, multi-model pipelines. The practical consequence is that there's no error propagation between stages, and no need to maintain and version multiple models.

Switching between tasks is accomplished simply by altering the input prompt, proving that a VLM can achieve competitive detection results compared to traditional detection models like DocLayout-YOLO. That's a meaningful shift: the same model weights handle everything, and you control what you get by changing a prompt string.

Architecture: built from scratch for documents

The architecture is a standard ViT-LLM design, but with deliberate choices that separate it from just fine-tuning a general-purpose VLM. Built upon a compact 1.7B LLM, dots.ocr provides faster inference speeds than many other high-performing models based on larger foundations.

Two components are worth understanding:

  • Vision Encoder (1.2B params, trained from scratch): Rather than reusing an image-centric encoder pretrained on natural photos, the team trained their own from scratch on document imagery. It handles native high-resolution inputs of up to 11 million pixels, which matters a lot for dense text and small-font documents.
  • Language Decoder (1.7B params, Qwen2.5-1.5B base):