Perception Programs Boost GPT-5 Mini Vision Accuracy by 45 Points Without Training

A team of researchers has published a paper arguing that multimodal language models struggle to reason over vision tools like depth estimators and optical flow because of how tool outputs are presented, not because of model size or missing capabilities. Their fix, Perception Programs (P²), rewrites raw pixel-level tool outputs into short, structured text summaries that the language model can actually parse. The technique is training-free, model-agnostic, and pushes accuracy on hard perception benchmarks by 20 to 45 percentage points across models ranging from 4B open weights to GPT-5 Mini.

The code is on GitHub, and because P² does not modify weights or require additional tool calls, it can drop into existing tool-use pipelines as a post-processing step.

The problem hiding inside tool-augmented vision

Modern multimodal LLMs are frequently paired with specialist vision tools (monocular depth estimators, optical flow networks, keypoint matchers, object detectors) to help them see things pixels alone cannot easily convey. Despite access to these tool-generated cues, MLLMs often fail to benefit fully from them. Existing approaches typically feed raw tool outputs into the model, but these dense, pixel-level representations are misaligned with the language-native reasoning strengths of LLMs, and the model falls back on language priors.

The authors demonstrate this concretely. When GPT-5 Mini is given a depth map and in-context examples and asked to spell out near-to-far ordering, it fails badly. The Kendall tau between the model's reconstructed ordering and ground truth drops rapidly toward zero as the depth grid gets finer, meaning the model essentially cannot read a depth map at high resolution even when it is right in front of it. Turning that same dense output into a Perception Program makes a general MLLM behave as if it can read the modality.