Every major large language model tested against a new standardised poker benchmark has failed to approach expert-level play, exposing persistent gaps in strategic reasoning under uncertainty.

The GTO Wizard Benchmark, introduced in a preprint posted to ArXiv, provides a public API and evaluation framework for testing AI agents in Heads-Up No-Limit Texas Hold'em (HUNL) — a two-player poker format long considered a demanding test of strategic reasoning, deception, and probabilistic inference. The benchmark measures agents against GTO Wizard AI, a superhuman poker agent built to approximate Nash Equilibria, the theoretical gold standard of optimal game strategy.

A Tougher Bar Than Any Previous Public Test

The benchmark displaces Slumbot, the champion of the 2018 Annual Computer Poker Competition and previously the strongest publicly accessible HUNL reference agent. According to the paper, GTO Wizard AI defeated Slumbot by $19.4 ± 4.1$ big blinds per 100 hands (bb/100) — a margin that poker professionals would consider decisive. By anchoring the benchmark to a stronger opponent, the researchers have raised the threshold substantially for any system hoping to claim competitive performance.

Poker evaluation presents a specific statistical challenge: variance. A single run of hands can mislead because luck plays a significant role over short samples. The benchmark addresses this by integrating AIVAT — a provably unbiased variance reduction technique — which, according to the authors, achieves equivalent statistical significance with ten times fewer hands than standard Monte Carlo evaluation. That reduction in required sample size makes the benchmark faster and cheaper to run without sacrificing rigour.

All models remain far below the baseline established by the benchmark, despite dramatic progress in LLM reasoning over recent years.

What the LLM Results Actually Show

The researchers ran a comprehensive zero-shot evaluation — meaning models received no task-specific training or fine-tuning — across several frontier systems, including GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, and Grok 4. Zero-shot conditions test whether a model's general reasoning capabilities transfer to a novel strategic environment without coaching.

The results were uniform in one respect: every model fell well short of the benchmark baseline. The paper does not publish exact per-model scores in the abstract but describes the shortfall as dramatic. That finding is notable given that several of these models have demonstrated strong performance on mathematical reasoning tasks, coding challenges, and formal logic problems — domains that superficially resemble the analytical demands of poker.

Qualitative analysis in the paper identifies two specific failure modes. The first is representation: how a model internally encodes the state of the game, including bet sizes, pot odds, and hand ranges. The second is reasoning over hidden states — the ability to make calibrated inferences about an opponent's concealed cards based on their actions. Both are core to competent poker play, and both remain areas where current LLMs show clear weaknesses, according to the authors.

Why Poker Remains a Hard Problem for Language Models

Poker is a game of imperfect information, which distinguishes it from challenges like chess or Go where both players see the full board. A competent poker player must maintain a probability distribution over all possible opponent hands, update it in real time as new cards are revealed and bets are made, and act in ways that balance exploiting predictable opponents with remaining unpredictable themselves.

This combination of probabilistic inference, multi-step planning, and strategic deception has historically been a frontier problem in AI. Libratus and Pluribus, purpose-built game-theory solvers developed by Carnegie Mellon University, cracked high-stakes human poker between 2017 and 2019 — but those systems were narrow specialists, not general-purpose reasoners. The question this benchmark implicitly asks is whether the general reasoning capabilities of modern LLMs can close that gap without specialised training.

The answer, for now, is no. But the authors note that the gap has narrowed meaningfully over recent model generations, suggesting the benchmark will serve as a useful longitudinal tracker of progress.

An Open Infrastructure for AI Reasoning Research

Beyond its immediate results, the benchmark's significance lies in what it offers to the research community. Its public API means any team can evaluate their system against a consistent, high-quality reference opponent without needing to build their own poker infrastructure. The AIVAT integration reduces the computational cost of statistically meaningful evaluation, lowering the barrier further.

The authors frame HUNL as a multi-agent system with partial observability — a formal problem class that extends well beyond poker. Autonomous systems navigating real-world environments, negotiation agents, and financial trading systems all share the same core challenge: acting optimally when critical information is hidden and opponents or environments respond strategically.

Progress on this benchmark, the paper argues, should correlate with broader advances in planning and reasoning that matter outside the card table.

What This Means

For researchers and developers, the GTO Wizard Benchmark establishes a rigorous, publicly accessible test that current frontier models cannot pass — making it a meaningful tool for tracking genuine progress in AI reasoning under uncertainty, rather than performance on saturated leaderboards.