Three of the world's leading large language models share near-identical cultural blind spots, systematically over-representing surface-level cultural markers while missing the deeper social values and priorities that native populations consider most important — according to a new study posted to ArXiv in April 2025.
The research introduces a structured method for measuring what the authors call "cultural alignment": not whether an AI can recite accurate facts about a culture, but whether its outputs reflect what people from that culture actually care about. Prior evaluations of cultural representation in AI have focused on diversity and factual correctness, leaving this more nuanced question largely unexamined.
How the Researchers Measured Cultural Fidelity
The team built their evaluation framework in two stages. First, they gathered open-ended survey responses from participants across nine countries, then used those responses to construct what they call Cultural Importance Vectors — essentially ranked lists of what people in each country consider culturally significant. These human-derived baselines served as the ground truth.
Second, the researchers designed a syntactically varied set of prompts to query Gemini 2.5 Pro, GPT-4o, and Claude 3.5 Haiku, generating what they call Cultural Representation Vectors for each model. Comparing the two sets of vectors revealed how closely each model's cultural output matched real human priorities.
All benchmark comparisons in this study are based on the researchers' own methodology and have not been independently verified.
All three models share systemic error signatures with a correlation above 0.97 — meaning their cultural failures are not random, but structurally alike.
A Western Lens That Sharpens With Distance
The findings point to a consistent pattern: alignment between model outputs and human expectations decreases as a country's cultural distance from the United States increases. In practical terms, the models perform reasonably well when representing cultures closer to Western norms, but diverge significantly when asked to represent cultures further from that baseline.
This Western-centric calibration is not unique to one model or one company. According to the study, all three frontier models — built by Google, OpenAI, and Anthropic respectively — exhibit the same directional bias. The researchers describe this as a "systemic error signature" with a correlation coefficient of ρ > 0.97 across all three systems.
That figure is significant: it suggests the problem is not an accident of individual training choices but a structural characteristic shared across the industry's leading products.
What the Models Get Wrong — and Right
The models tend to over-index on visible, easily catalogued cultural markers — food, festivals, traditional dress — while underweighting the social structures, value systems, and community priorities that native respondents ranked as more central to their cultural identity. This distinction matters because surface-level representation can create an appearance of cultural awareness while missing what actually defines a culture from the inside.
The researchers stop short of attributing the bias to any single cause, but the pattern is consistent with what critics of AI training pipelines have long argued: that internet-scraped training data over-represents English-language, Western-origin content, shaping model behaviour in ways that persist even when models are queried about non-Western subjects.
A New Benchmark for Cultural AI Evaluation
The study's broader contribution is methodological. By establishing human-derived importance vectors as a baseline and comparing them against model-derived representation vectors, the team offers a replicable framework that goes beyond asking whether an AI's cultural claims are factually accurate.
This distinction — between factual accuracy and authentic prioritisation — is significant. A model could correctly state that a given country celebrates a particular holiday while still fundamentally misrepresenting that culture by treating the holiday as more important than the familial or religious values that actually animate it.
The framework was applied to only three models and nine countries in this study, which limits the immediate scope of the findings. Expanding the country set and including additional models would strengthen the conclusions, and the authors' methodology is designed to be extensible.
The study does not address whether model fine-tuning or retrieval-augmented approaches might reduce the bias detected — that remains an open question for follow-on research.
What This Means
For organisations deploying AI in global or multicultural contexts, this research provides concrete evidence that cultural representation errors are not model-specific accidents but a shared, measurable structural problem — one that existing diversity metrics are unlikely to catch.