Skip to main content
Back

K-BrowseComp Exposes How GPT-5.5 Drops From 84% to 45% on Korean Web

K-BrowseComp reveals that even GPT-5.5 tops out at 46% on Korean web browsing tasks, while Korea's own government-funded models score near zero

Paper, Full Access
  • K-BrowseComp is a new 400-question benchmark for web-browsing agents grounded entirely in Korean web contexts, built by researchers from KAIST, SNU, and CMU.
  • GPT-5.5 tops the leaderboard at just 45.67%, a massive drop from its 84.4% on the English BrowseComp; DeepSeek-V4-Pro falls from 83.4% to 30%.
  • Korea's own government-funded models score 0% to 10.33%, with Kakao's Kanana model failing to complete any browsing runs due to tool-call protocol failures.
  • The core finding: models fail after retrieving relevant Korean evidence, not before -- they cannot maintain candidate sets, constraints, and entity roles across multi-step trajectories.
  • A 100-problem synthetic split was generated using an AI agent targeting nine categorized failure modes; the strongest model scores only 26% on it.
  • The dataset and code are publicly released under MIT License, built on top of Perplexity's search_evals framework.

AI agents are getting remarkably good at browsing the English web. OpenAI's BrowseComp, a benchmark of 1,266 hard research questions requiring multi-hop web navigation, sees top models like GPT-5.5 and DeepSeek-V4-Pro score above 80%. But swap the web for Korean-language sources, and those same models crater. A new benchmark called K-BrowseComp, built by researchers from KAIST, Seoul National University, Chung-Ang University, and Carnegie Mellon, exposes just how badly the field has neglected non-English agentic evaluation.

The benchmark consists of 400 problems grounded entirely in Korean web contexts. The core 300-question set, called K-BrowseComp-Verified, was hand-crafted and validated by 15 native Korean speakers. The strongest evaluated models achieve only 30.00-45.67% on K-BrowseComp-Verified. That is a dramatic drop from their reported performance on the original English BrowseComp. GPT-5.5 and DeepSeek-V4-Pro obtain 45.67% and 30.00%, respectively, which is a substantial drop from their reported performance on the original BrowseComp (84.4% and 83.4%, respectively).

A gap hiding in plain sight

BrowseComp fills an important gap in agent evaluation by specifically testing the persistence and creativity required for real-world web research, where information is often buried deep within websites or requires synthesizing clues across multiple sources. But that benchmark, and the agents trained against it, are almost entirely English-centric. While leading frontier models from the US and China are shifting their evaluation focus toward compositional agentic evaluation, the Korean AI community remains largely anchored to static benchmarks, and Korean agentic benchmarks are still virtually nonexistent.

This matters for two distinct reasons. First, there is an AI sovereignty angle: Korean users querying local government portals, entertainment databases, or educational records are poorly served by agents optimized for English search conventions. Second, Korean web content provides a principled out-of-distribution testbed for the broader research community, since frontier models are increasingly saturating English-centric benchmarks and the line between memorization and genuine reasoning is blurring.

What the benchmark actually tests

Each question is designed to have a single short answer that is stable over time and supported by publicly accessible web evidence. Every problem falls into one of two reasoning formats:

  • Multi-hop: The agent must find an intermediate fact, then use it to retrieve the next piece of evidence, chaining at least four steps. Example: identify a poem discussed in a 2018 Korean college entrance exam, find the poet's tenth published collection, then return the title of the thirteenth poem in that collection.