KRAFTON AI's Orak Drops LLMs Into 12 Real Video Games
KRAFTON and NVIDIA built a 12-game benchmark spanning every major genre to test how LLM agents actually play real video games.
PRO- KRAFTON and NVIDIA released Orak, a 12-game LLM agent benchmark spanning all six major game genres.
- Plug-and-play MCP interface lets you swap LLM backbones and agentic modules (reflection, planning, memory, tools) with config changes.
- Gemini-2.5-pro tops the leaderboard on average rank, with GPT-5 and GPT-4o close behind across 15 tested models.
- Fine-tuning dataset on Hugging Face transfers gameplay skills to WebShop and Math500 tasks.
- Image-only inputs dramatically underperform text; real-time StarCraft II wipes out every model except GPT-4o.
- Small open-source LLMs (under 8B) score near zero on complex games and don't benefit from agentic scaffolding.
A team from KRAFTON AI, NVIDIA, Seoul National University, and the University of Wisconsin-Madison has released Orak, a benchmark that drops language model agents into twelve commercial video games and measures how well they can actually play them. The name comes from the Korean word for arcade, and the games were picked both for their cultural weight in the industry and for their spread across genres.
Existing game benchmarks for LLMs tend to rely on text-only adventures or 2D grid worlds, skip the agentic scaffolding that matters in practice, and offer nothing to fine-tune on. Orak tries to fix all three problems at once, and the paper ships with a leaderboard, a battle arena, ablations, and a supervised fine-tuning dataset on Hugging Face.
Twelve games, six genres, one interface
The roster is deliberately eclectic and covers every major genre: Street Fighter III, Super Mario, Ace Attorney, Her Story, Pokémon Red, Darkest Dungeon, Minecraft, Stardew Valley, StarCraft II, Slay the Spire, Baba Is You, and 2048. Each title stresses different cognitive skills, so the benchmark ends up probing seven capability axes at once: rule following, logical reasoning, spatial reasoning, long-text understanding, long-term planning, error handling, and odds handling.
To keep evaluation reproducible as models evolve, the team built the whole thing on top of the Model Context Protocol. Each game environment and agentic module operates as an independent MCP server, exposing game mechanics (retrieving state, executing steps) or agentic strategies (reflection, planning) as callable tools. Swapping backbones or agent strategies becomes a config change rather than a rewrite.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
