Skip to main content
Back

MMG2Skill Turns Raw Internet Guides Into Self-Improving AI Agent Skills

MMG2Skill turns messy web tutorials into self-improving agent skills, beating vanilla VLM baselines by up to 25 percentage points across GUI, Minecraft, and card games

  • Core idea: MMG2Skill converts messy, human-oriented web guides into structured, self-improving agent skills without fine-tuning any model weights.
  • Key finding: Feeding raw guides directly to agents hurts performance; structured compilation + trajectory-driven revision is what actually helps.
  • Results: +12.8 to +25.3 percentage point gains over vanilla VLM baselines across all six tested model backbones and three domains.
  • Benchmark: MMG2Skill-Bench covers 130 tasks across desktop GUI (OSWorld), Minecraft, and card games (Doudizhu/Mahjong via RLCard.
  • Efficiency: Analyzer-based early stopping saves 25%-53% of inference attempts on success-inferable tasks.
  • Limitation: Guide selection from large corpora is unsolved; no code release yet; tested only on VLM backbones.

The internet is full of how-to guides: YouTube walkthroughs, wiki pages, community tutorials. Humans use them all the time to learn new software or pick up a game. But hand one of those guides directly to an AI agent and it often performs worse than if you had given it nothing at all. A new paper from a team of researchers introduces MMG2Skill, a framework that bridges this gap by converting raw, human-oriented guides into structured, self-improving agent skills.

The gap nobody was measuring

The web holds abundant procedural knowledge for long-horizon tasks, but such knowledge is often multimodal, heterogeneous, noisy, and implicitly assumes human executors, making it difficult to use directly as the skills required by agents. Think about a Minecraft wiki page: it assumes you know what a crafting table looks like, that you have hands, and that you understand spatial metaphors like "place in the top-left slot." An agent operating through an API has none of that implicit context.

Recent agent systems augment LLMs with reusable behavior stored as language memories, executable programs, or skill libraries. But the skills in these systems are typically expert-provided, model-generated, or discovered through the agent's own experience. As a result, there is no systematic evidence about whether public guide material can become agent-executable skills, and no framework that can make this conversion without relying on benchmark scores during revision.

To bridge the gap between human-oriented guides and agent-executable skills, the authors formalize this problem as guide-to-skill learning: converting in-the-wild guides into executable skills and continuously improving them from trajectories observable to the agent. The key constraint: the system must improve itself using only what the agent can actually see during execution, not privileged benchmark labels.

A benchmark built for the real world

To address these gaps, the team introduces MMG2Skill-Bench, the first benchmark for guide-to-skill learning from in-the-wild multimodal guides. The benchmark spans desktop GUI control, open-ended game play, and strategy tasks, enabling evaluation beyond a single environment or action interface.