MIT's Indica Benchmark Catches Frontier AI Defaulting to North Indian Answers
A new benchmark exposes how eight frontier LLMs treat India as culturally uniform, defaulting to Central and North Indian practices when generating cultural answers.
ProTakeaways
- Indica is the first benchmark testing sub-national cultural commonsense, with 1,630 region-specific QA pairs across five Indian regions.
- Only 39.4% of questions achieved consensus across all five regions, showing Indian culture is predominantly regional.
- Eight frontier LLMs scored just 13.4% to 20.9% fully correct on region-specific questions.
- Models over-selected Central India by 37% and North India by 21% versus expected uniform selection.
- Error analysis showed 89% of partially correct answers were overexplaining from a generic cultural template.
- Full dataset and code available on GitHub and HuggingFace.
A team from Emory, Brock, and MIT has released a benchmark that quietly demolishes a comfortable assumption baked into most cultural evaluations of language models: that a country is a coherent cultural unit. Indica, short for Indian Cultural commonsense Inventory with Cross-regional Answers, tests whether frontier LLMs understand that cultural commonsense in India varies wildly by region, and finds that they mostly don't.
Only 39.4% of questions in the benchmark achieve consensus across India's five regions, confirming that cultural commonsense in India is predominantly regional. When eight frontier models were asked region-specific questions, they scored between 13.4% and 20.9% on fully correct answers, capturing broad concepts but missing local knowledge. Forced to choose between regional practices without geographic context, they systematically defaulted to Central and North Indian answers.
The monolith problem
Existing cultural benchmarks like CultureBank and CulturalBench treat each country as a single cultural unit. That works for questions like which side of the road to drive on, but breaks down for a country with 28 states, 8 union territories, and 22 official languages. Prior Indian benchmarks such as MILU, SANSKRITI, and IndicQuest have focused on factual knowledge from textbooks rather than the messier terrain of everyday practice.
Indica is the first attempt to test whether that terrain is uniform, and it isn't. The team collected human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs.

Building a benchmark that travels
The construction pipeline is worth understanding because it's designed to be replicated for any culturally diverse nation. The team grounded domain selection in the Outline of Cultural Materials (OCM), an established anthropological taxonomy organizing cultural knowledge into 90+ major categories and 700+ subcategories, widely used in cross-cultural research. The eight domains covered: