---
title: "MMSkills Doubles Small AI Agents' Success Rate on Desktop Tasks"
subtitle: "MMSkills gives visual agents reusable, screenshot-grounded skill packages that more than double small model success rates on real desktop and game benchmarks"
url: "https://alphasignal.ai/news/mmskills-doubles-small-ai-agents-success-rate-on-desktop-tasks"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-05-01T00:00:00.000Z"
updated: "2026-06-09T05:03:34.273Z"
type: "Paper"
topics:
  - "Agents"
  - "Development"
subtopics:
  - "Computer Use"
  - "Code Agents"
  - "Code Agents"
upvotes: 3134
source_url: "https://arxiv.org/abs/2605.13527"
comments: 0
truncated: true
---

# MMSkills Doubles Small AI Agents' Success Rate on Desktop Tasks

> MMSkills gives visual agents reusable, screenshot-grounded skill packages that more than double small model success rates on real desktop and game benchmarks

Teaching an AI agent to click through a spreadsheet or navigate a Minecraft inventory is hard enough. Teaching it to **remember how it did that last time** \-- and recognize the right moment to reuse that knowledge -- has been even harder. A new framework from researchers at Shanghai Jiao Tong University, Xiaohongshu Inc., and Southeast University proposes a direct fix: give agents skill packages that bundle written procedures with actual screenshots, so they can match what they see on screen to what they already know how to do.

The framework is called [MMSkills](https://arxiv.org/abs/2605.13527), and it introduces what the authors call **multimodal procedural knowledge** \-- a formalization of the idea that reusable agent skills for visual environments must include not just instructions, but also visual evidence of what the right state looks like, what progress looks like, and what failure looks like.

## The gap text-only skills leave open

Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. That works fine when the task can be fully described in words. But visual agents live in a different world. For visual agents, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next.

The concrete failure mode is easy to picture: a desktop agent may know the correct sequence of menu clicks to insert a chart, but fail to recognize that the wrong spreadsheet tab is currently active. A text skill that says "click Insert > Chart" is useless if the agent cannot tell it is looking at the wrong sheet. Reliable desktop automation has long come with a hidden tax: the more complex the software environment, the larger -- and more expensive -- the model required to run it. MMSkills challenges that assumption directly.

![MMSkills example showing multimodal skill package for chart creation in a spreadsheet, with state cards and visual evidence](https://arxiv.org/html/2605.13527v3/x1.png) 

## What an MMSkill actually contains

Each MMSkill is a structured package with four components: a compact descriptor, a reusable textual procedure, a set of **runtime state cards**, and a set of **multi-view keyframe bundles**. The state cards are the key innovation. Each one is not just a screenshot caption -- it is an agent-facing decision node that encodes:

* **When to use** this procedural step

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [MMSkills: Towards Multimodal Skills for General Visual Agents (arXiv HTML)](https://arxiv.org/html/2605.13527)
- [MMSkills: Towards Multimodal Skills for General Visual Agents (arXiv abstract)](https://arxiv.org/abs/2605.13527)
- [Visual State Cards in AI Agent Skills More Than Double Small Model Success Rates (TechTimes)](https://www.techtimes.com/articles/316809/20260519/visual-state-cards-ai-agent-skills-more-double-small-model-success-rates-real-desktop-tasks.htm)
- [MMSkills GitHub Repository](https://github.com/DeepExperience/MMSkills)
- [MMSkills HuggingFace Dataset](https://huggingface.co/datasets/zhangkangning/mmskills)