A new benchmark called SEA-Eval has revealed that state-of-the-art AI agents can appear equally competent under standard tests while hiding efficiency gaps of up to 31.2 times in computational cost — a disparity only visible when agents are evaluated across sequences of tasks over time.

Most AI agent benchmarks today treat each task as an isolated episode: the agent starts fresh, completes a task, and receives a score. This approach, according to the researchers behind SEA-Eval, overlooks something fundamental — whether an agent actually gets better with experience. The paper, published on arXiv in April 2025, formalises a concept called the Self-Evolving Agent (SEA) paradigm and introduces the first evaluation framework built specifically around it.

Why Episodic Benchmarks Miss the Point

Current large language model (LLM)-based agents face two structural problems that episodic benchmarks cannot detect. The first is reliance on a fixed set of tools — agents cannot acquire new capabilities as their environment changes. The second is what the researchers call "episodic amnesia": agents retain no memory or learning between tasks, meaning every new problem is approached as if it were the first.

Identical success rates mask up to 31.2 times differences in token consumption and divergent evolutionary trajectories under sequential analysis.

These limitations matter because real-world deployments rarely consist of isolated, self-contained tasks. An agent managing a software development workflow, for example, encounters related problems repeatedly — and an agent that learns from earlier attempts should, in theory, handle later ones faster and more cheaply.

What SEA-Eval Actually Measures

SEA-Eval evaluates agents across two dimensions. The first is intra-task execution reliability — essentially, can the agent complete individual tasks correctly? This maps to the success rate metric familiar from existing benchmarks. The second, and novel, dimension is long-term evolutionary performance: does the agent improve in efficiency and strategy as it processes more tasks sequentially?

To measure this, SEA-Eval organises tasks into sequential streams rather than isolated episodes. It then tracks two metrics over time: Success Rate (does the agent complete the task?) and Token Consumption (how much computational work does it take?). Token consumption serves as a proxy for efficiency — an agent that learns should theoretically need fewer tokens to handle familiar problem types as it accumulates experience.

The benchmark introduces two derived concepts: "evolutionary gain," which quantifies improvement in efficiency over time, and "structural stability," which measures whether an agent's performance trajectory is consistent or erratic.

The Bottleneck Hidden in Plain Sight

The empirical results are the paper's most striking contribution. When the researchers applied SEA-Eval to current state-of-the-art agent frameworks, they found that frameworks achieving identical success rates could diverge dramatically in token consumption — by as much as 31.2 times. Under conventional benchmarks, these agents would be rated as equivalent. Under sequential analysis, they reveal entirely different evolutionary trajectories.

This has direct practical consequences. Token consumption translates to computational cost, which translates to money and energy. An enterprise deploying an AI agent at scale that consumes 31 times more tokens than a comparable alternative — for the same task success rate — faces a hidden cost that standard benchmarks would never flag.

The paper also notes that divergent evolutionary trajectories suggest agents are not just slower, but qualitatively different in how they handle sequential experience. Some frameworks may plateau quickly; others may degrade under repeated task exposure.

A Formal Definition for a Fuzzy Concept

Beyond the benchmark itself, the paper makes a conceptual contribution by grounding the SEA paradigm in what the authors call "digital embodiment" — the idea that an agent exists persistently within a digital environment, accumulating state and experience the way a physical agent accumulates knowledge of its surroundings. Previous uses of the SEA concept in the literature lacked a rigorous formal definition, according to the authors.

This formalisation matters because it sets clear criteria for what would actually constitute a self-evolving agent, rather than treating any agent with memory features as automatically qualifying. The authors distinguish between agents that merely store past outputs and agents that genuinely restructure their strategies based on accumulated experience — a distinction existing benchmarks collapse entirely.

The benchmark, according to the paper, is designed to be the first scientific foundation for measuring progress toward the latter — agents that evolve rather than simply remember.

What This Means

SEA-Eval reframes what "better" means for AI agents: success rate alone is no longer sufficient, and developers building or selecting agent frameworks now have a concrete tool for measuring whether their systems actually learn from experience — and at what cost.