Microsoft's OmniParser Hits 25K Stars Turning Any LLM Into a Computer Agent

Microsoft's OmniParser project keeps compounding. The repository has crossed 25,000 stars, and the team has just slotted in a new YOLOv9-E interactive region detector, the latest step in a roadmap that already includes V2 checkpoints, an OmniTool wrapper for driving a Windows 11 VM, and support for local trajectory logging so you can build your own agent training pipeline.

If you've been ignoring screen parsers because you assumed a big vision-language model could just look at pixels directly, this repo is the counterargument. OmniParser tokenizes UI screenshots from pixel space into structured elements that are interpretable by LLMs, letting the model do retrieval-based next-action prediction over a set of parsed interactable elements.

What actually shipped

The recent addition is a YOLOv9-E interactive region detector, with inference-only weights available through a Hugging Face PR. That sits on top of the V2 stack Microsoft released earlier, which is the version most people should be using today.

Under the hood, V2 is a two-model pipeline: a fine-tuned YOLOv8 icon detector paired with a Florence-2-based caption model, where the detector localizes interactive regions and the captioner describes their function. Critically, the parsing loop itself does not call an LLM, which is what keeps latency usable for agent execution.