---
title: "KRAFTON AI's Orak Drops LLMs Into 12 Real Video Games"
subtitle: "KRAFTON and NVIDIA built a 12-game benchmark spanning every major genre to test how LLM agents actually play real video games."
url: "https://alphasignal.ai/news/krafton-ai-s-orak-drops-llms-into-12-real-video-games"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-14T00:00:00.000Z"
updated: "2026-08-23T23:25:46.511Z"
type: "Paper"
topics:
  - "Agents"
  - "Benchmarks"
subtopics:
  - "Agent Frameworks"
  - "Mcp"
upvotes: 3452
source_url: "https://arxiv.org/abs/2506.03610"
comments: 0
truncated: true
---

# KRAFTON AI's Orak Drops LLMs Into 12 Real Video Games

> KRAFTON and NVIDIA built a 12-game benchmark spanning every major genre to test how LLM agents actually play real video games.

A team from [KRAFTON AI](https://www.krafton.ai/en/finding-an-llm-thats-truly-good-at-gaming-the-birth-of-the-orak-benchmark/), NVIDIA, Seoul National University, and the University of Wisconsin-Madison has released [Orak](https://github.com/krafton-ai/Orak), a benchmark that drops language model agents into twelve commercial video games and measures how well they can actually play them. The name comes from the Korean word for arcade, and the games were picked both for their cultural weight in the industry and for their spread across genres.

Existing game benchmarks for LLMs tend to rely on text-only adventures or 2D grid worlds, skip the agentic scaffolding that matters in practice, and offer nothing to fine-tune on. Orak tries to fix all three problems at once, and the paper ships with a leaderboard, a battle arena, ablations, and a supervised fine-tuning dataset on Hugging Face.

![Orak benchmark architecture overview with 12 games, MCP interface, and evaluation leaderboards](https://arxiv.org/html/2506.03610v3/figure1_ORAK_overview.png) 

## Twelve games, six genres, one interface

The roster is deliberately eclectic and covers every major genre: Street Fighter III, Super Mario, Ace Attorney, Her Story, Pokémon Red, Darkest Dungeon, Minecraft, Stardew Valley, StarCraft II, Slay the Spire, Baba Is You, and 2048\. Each title stresses different cognitive skills, so the benchmark ends up probing seven capability axes at once: rule following, logical reasoning, spatial reasoning, long-text understanding, long-term planning, error handling, and odds handling.

![Radar charts comparing required LLM capabilities across 12 games in Orak](https://arxiv.org/html/2506.03610v3/figure3_LLM_capabilities.png) 

To keep evaluation reproducible as models evolve, the team built the whole thing on top of the Model Context Protocol. Each game environment and agentic module operates as an independent MCP server, exposing game mechanics (retrieving state, executing steps) or agentic strategies (reflection, planning) as callable tools. Swapping backbones or agent strategies becomes a config change rather than a rewrite.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games](https://arxiv.org/html/2506.03610)
- [krafton-ai/Orak GitHub repository](https://github.com/krafton-ai/Orak)
- [KRAFTON/Orak Dataset on Hugging Face](https://huggingface.co/datasets/KRAFTON/Orak)
- [Orak Leaderboard](https://krafton-ai.github.io/orak-leaderboard/)
- [Finding an LLM That's Truly Good at Gaming - KRAFTON AI](https://www.krafton.ai/en/finding-an-llm-thats-truly-good-at-gaming-the-birth-of-the-orak-benchmark/)