Autonomous AI agents can sustain meaningful improvements in marketing engagement without continuous human oversight, according to an 11-month longitudinal case study published on ArXiv, though human-led strategy phases still produced the strongest results.
The research addresses a practical question that has divided marketing technologists: how much human supervision does an AI personalisation system actually need to keep performing over time? Customer Relationship Management (CRM) has traditionally depended on marketers manually tuning rule-based messaging — a process that doesn't scale easily across millions of users. Autonomous learning systems promise to close that gap, but until now there has been limited real-world evidence on whether performance gains hold once human hands come off the wheel.
What the Study Actually Measured
The researchers examined a live consumer application using agentic infrastructure — AI systems capable of taking actions, making decisions, and adapting strategies within defined parameters — to personalise marketing messages across a large user base. The study split the observation window into two consecutive phases. In the active phase, human marketers directly curated content, selected audience segments, and shaped messaging strategies. In the passive phase, agents operated entirely autonomously from a fixed library of pre-existing components, with no new human input.
Human intervention drives strategic initialisation and discovery, yet autonomous agents can ensure the scalable retention and preservation of performance gains.
The key metric tracked across both phases was engagement lift — the improvement in user engagement attributable to personalisation, measured relative to a baseline. The paper does not disclose the specific application or the company behind it, and the benchmark figures cited are drawn from the authors' own system measurements rather than independently verified third-party audits.
Humans Set the Ceiling, Agents Hold the Floor
The results showed a clear division of labour between human and machine contributions. Active human management generated the highest relative lift in engagement metrics — marketers exploring new content, testing audience strategies, and refining approaches in real time produced gains that autonomous agents alone did not match. This suggests that creative discovery and strategic experimentation still benefit significantly from human judgment.
However, the more striking finding for practitioners is what happened during the passive phase. Once human curation stopped, the autonomous agents did not simply degrade toward baseline performance. Instead, they sustained a positive engagement lift throughout the passive period by continuing to apply and optimise from the fixed component library left by the active phase. The system preserved what had been learned rather than forgetting it.
The authors describe this as a symbiotic model: humans initialise and explore, agents retain and scale. Neither operates optimally in isolation, but the combination produces durable results at a scale that purely human management cannot match.
Why the 'Human-in-the-Loop' Question Matters
The phrase "human-in-the-loop" has become something of a reflex in AI deployment discussions, often invoked without precision about what form oversight should actually take or how frequently it needs to occur. This study provides empirical grounding to that debate within the specific domain of marketing personalisation.
If autonomous agents can sustain performance for extended periods — in this case, across the passive phase of an 11-month window — then the argument for continuous human monitoring weakens. That has direct implications for how organisations staff and structure their marketing operations, and for how they evaluate the cost-benefit trade-off of deploying agentic systems versus maintaining large manual optimisation teams.
It also raises questions the study does not fully resolve. The passive phase relied on a fixed library of components assembled during the active phase — meaning the agents were applying and refining existing strategies rather than generating entirely novel ones. Whether performance would continue to hold over a significantly longer passive period, or whether it would eventually stagnate without fresh human input, remains an open question the authors acknowledge.
Limits of a Single Case Study
The research is a single case study of one undisclosed application, which limits how broadly the conclusions can be generalised. Different product categories, user demographics, or content types may produce different dynamics between active and passive phases. The engagement metrics used are self-reported by the system rather than externally audited, and the paper does not provide granular figures on the size of the lift differences between phases.
That said, the study's value lies less in its specific numbers and more in its longitudinal structure. Eleven months is a meaningful timeframe in a domain where most AI performance evaluations are measured over weeks, and the sequential design — active followed immediately by passive — provides a cleaner comparison than many real-world studies can achieve.
Future research could usefully examine what happens when passive periods extend beyond a year, whether periodic "reactivation" of human curation can restore peak performance, and how the composition of the initial component library affects long-term autonomous performance.
What This Means
For organisations deploying AI in marketing, this study suggests that investing heavily in the human-led initialisation phase pays dividends that autonomous agents can then preserve at scale — meaning the question is less "humans or AI" and more "when and how much of each."