Generative AI systems can cross a calculable internal threshold that causes their output to flip from reliable legal reasoning to authoritative-sounding fabrication, according to a new paper published on arXiv. The researchers argue this failure mode is foreseeable enough to carry direct professional and legal consequences for attorneys who use these tools.
The paper, submitted to arXiv's CS.AI section, draws on physics-based analysis of the transformer architecture—the mechanism underlying most large language models in commercial use—to argue that what the industry commonly labels "hallucination" is not purely random noise. Instead, the authors contend, an AI system's internal state can reach a deterministic tipping point, after which it generates invented case citations, fabricated statutes, and fictional judicial holdings with the same surface confidence it displays when producing accurate output.
Why "Hallucination" Is the Wrong Frame
The hallucination label has been widely adopted by both the technology industry and legal commentators, and it carries implicit meaning: unpredictable, sporadic, hard to anticipate. The arXiv paper challenges that framing directly. According to the authors, the transformer's core mechanism means that fabrication risk is not an anomalous glitch but a foreseeable consequence of the technology's design.
"Fabrication risk is not an anomalous glitch but a foreseeable consequence of the technology's design, with direct implications for the evolving duty of technological competence."
This distinction matters enormously in a legal context. If AI fabrication is random and unforeseeable, it functions more like an act of God—difficult to assign liability for. If it is foreseeable and structurally inherent, it shifts the analysis toward negligence and professional responsibility frameworks that courts and bar associations already have tools to apply.
What the Transformer Threshold Means in Practice
The paper walks through a simulated brief-drafting scenario to illustrate how the failure mode manifests. An attorney using a generative AI tool to research precedent or draft a motion might receive output that includes plausible-sounding but entirely invented case names, docket numbers, and holdings. The fabrications are not flagged by the system; they arrive formatted and confident, embedded within otherwise coherent legal reasoning.
Several high-profile cases have already demonstrated the real-world consequences. In 2023, a New York federal court sanctioned attorneys who filed a brief containing six fictitious case citations generated by ChatGPT, finding the lawyers had failed to verify the AI's output. That case, Mata v. Avianca, became a landmark moment for the legal profession's reckoning with AI tools, prompting bar associations and courts across multiple jurisdictions to issue guidance on AI use in legal practice.
The arXiv paper situates these incidents within a broader structural argument: they were not unlucky edge cases but predictable outputs of a system operating exactly as its architecture allows.
Professional Duty of Competence Under Pressure
In the United States, the American Bar Association's Model Rule 1.1 on competence has been interpreted—including in a 2012 formal opinion and subsequent updates—to require that attorneys keep pace with relevant technology. Several state bars, including those in California and Florida, have issued specific guidance applying competence obligations to AI-assisted legal work. None of these instruments are binding federal law; they are professional conduct rules enforced through state bar disciplinary mechanisms.
The paper's authors argue that if fabrication is a foreseeable design characteristic rather than a random fault, the competence standard requires attorneys to implement active verification protocols—not merely to exercise general caution. According to the paper, the appropriate response is to replace the "black box" mental model of AI systems with workflows grounded in how these tools actually fail.
The enforcement mechanism here is professional discipline: bar complaints, sanctions, and malpractice claims. Courts also retain inherent authority to sanction attorneys for filing false or misleading documents, regardless of whether AI was involved in their preparation. No specific AI liability statute governs this space in most U.S. jurisdictions, meaning existing professional responsibility doctrine is currently doing most of the legal work.
Courts and Regulators Are Still Catching Up
Several U.S. federal district courts have adopted local rules requiring attorneys to disclose AI use in filings and certify that AI-generated content has been verified. These rules vary significantly by jurisdiction and are not uniform. The European Union's AI Act, which entered force in 2024, classifies certain AI applications in the administration of justice as high-risk, requiring conformity assessments and human oversight—but its application to private legal practice rather than judicial decision-making is less clearly defined.
The arXiv paper does not propose specific regulatory language, but its framing carries regulatory implications. If fabrication risk is calculable and threshold-based, regulators could in principle require AI vendors selling to legal markets to disclose the conditions under which their systems are more likely to tip into fabrication—a form of technical transparency that current disclosure regimes do not yet mandate.
The paper's authors propose that legal professionals and courts adopt verification protocols designed around the actual failure architecture of transformer models, rather than general-purpose caution borrowed from earlier software reliability frameworks.
What This Means
If the paper's core argument holds—that AI fabrication in legal contexts is structurally foreseeable rather than random—attorneys who fail to verify AI-generated research face heightened exposure under existing professional competence rules, and regulators have a stronger technical basis for mandating specific oversight requirements rather than issuing advisory guidance alone.