Skip to main content
Back

MGSD Closes the Visual Planning Gap by 19% Without Symbolic Inputs

MGSD teaches vision-language models to plan spatially by distilling symbolic knowledge into visual reasoning, closing an 18-19% performance gap at inference time without any symbolic inputs.

  • Core result: MGSD improves visual spatial planning by ~19% on 4B and 8B VLM backbones over baseline, with no symbolic inputs at inference.
  • The problem: VLMs face a dual bottleneck in visual planning -- recovering scene structure from pixels AND reasoning over it for multi-step plans.
  • The method: A two-stage framework: cold-start grounding for perception, then on-policy self-distillation using a symbolic-state privileged teacher.
  • Key innovation: Symbolic state data is used only during training; the final model plans purely from visual input, closing the gap to text-based upper bounds.
  • Benchmark: Evaluated on the VSP benchmark (Maze Navigation + Blocks World), which isolates perception and reasoning sub-skills separately.
  • Code available: Framework is open-source at github.com/Oranger-l/MGSD, compatible with standard open-weight VLM backbones.

Ask a state-of-the-art vision-language model to navigate a maze from a screenshot, or rearrange blocks to match a target configuration, and it will likely fail -- not because it lacks raw intelligence, but because it is facing two compounding problems at once. It has to figure out the scene structure from raw pixels, and then reason over that inferred structure to produce a valid multi-step plan. A new paper introduces MGSD (Modality-Gap-Aware Self-Distillation), a training framework that tackles both bottlenecks simultaneously, delivering gains of roughly 19% on visual planning benchmarks while keeping inference purely visual.

The gap that nobody talks about

There is a well-known divide between how models handle text versus images. But in the planning domain, this gap has a very specific shape. One understudied capability in VLMs is visual spatial planning -- the ability to comprehend the spatial arrangements of objects and devise action plans to achieve desired outcomes in visual scenes. The trouble is that spatial planning is not one hard problem, it is two hard problems stacked on top of each other.

The MGSD paper calls this the perception-reasoning modality gap: visual planning requires a model to first recover latent state structure from pixels (where are the walls, the blocks, the obstacles?) and then reason over that recovered structure to produce valid actions. Symbolic planning, by contrast, hands the model an explicit list of objects and constraints. Both open-source and private VLMs fail to generate effective plans for even simple spatial planning tasks, with evaluations revealing fundamental deficiencies in visual perception and bottlenecks in reasoning abilities.

Diagram showing perception and reasoning gaps between base model, SFT, and MGSD with accuracy metrics

The result is a frustrating ceiling: give a model the same planning problem as text (symbolic state) and it performs dramatically better than when given an image. The gap between these two modes is the target MGSD is designed to close.