The Physics of the Hallucination: Why AI Can’t Help But Dream

The 110,000-Word Stress Test

A few years ago, I finished a high-fantasy novel—a sprawling, 350-page manuscript of roughly 110,000 words. Because of my professional background, I was written under a pseudonym, but the structural complexity was anything but fictional. It featured a rigid, "hard" magic system based on technical rules, dozens of characters, and multiple arcs spanning several cities.

My concern was continuity. In a 1,500-page series, a single contradiction in Chapter 2 can collapse the logic of Chapter 20. I turned to AI to act as my Master Auditor. My plan was to feed the entire book in and have the AI generate a master index of "truth-bits"—distilling 8,000-word chapters into 1,500-word summaries of pure fact.

The Five-Minute Miracle

I uploaded the entire manuscript. In less than five minutes, the AI spat out a 15,000-word summary. I read the summary of Chapter 1 and was floored. It didn't just summarize the plot; it understood the subtle emotional foreshadowing and the technical nuances of the magic system.

This is what researchers call the Fluency Heuristic—the psychological trap where a human assumes that because a machine writes with professional-grade polish, its underlying data must be accurate. Because it "got" the subtext of the first fifty pages, I instinctively lowered my guard for the next three hundred.

The Ghost in the Manuscript

I flipped to the summary of Chapter 10. The quality of the prose was still fabulous, but the utility had vanished. The AI was describing "Phantom Characters" I never created and plot points I never envisioned. By the final quarter of the book, the AI wasn't summarizing my novel anymore; it was writing its own.

But  I didn't just walk away frustrated. I asked the machine: "What happened?"

The AI’s answer was candid and self critical, but I didn't stop there. I began digging into the emerging research from institutions like Stanford and NYU, and what I found was startling. I wasn't witnessing a "glitch." I was witnessing the fundamental architecture of Large Language Models (LLMs) at work.

The Three Pathologies of Artificial Intelligence

Through my extensive use and subsequent research, I discovered three recurring phenomena that every attorney must understand before they sign a signature block on an AI-assisted brief.

1. Sycophancy (The People-Pleaser)

We often mistake AI for a search engine, but it is actually closer to an incredibly eager, overachieving junior associate. Technical research into Sycophancy shows that models are often trained via RLHF (Reinforcement Learning from Human Feedback) to prioritize "helpfulness" over "truth."

As noted by researchers at OpenAI and NYU, these models are mathematically rewarded for providing an answer that satisfies the user’s prompt. In my case, the AI didn't want to admit it was overwhelmed by 110,000 words of data. It "predicted" a helpful-sounding ending because, in its training, "I don't know" is often treated as a failure.

2. "Lost in the Middle" (The Attention Deficit)

Stanford researchers published a seminal paper titled "Lost in the Middle," which identified a U-shaped performance curve in how AI processes long documents.

Models exhibit high accuracy at the very beginning (Primacy) and the very end (Recency) of a prompt. However, as the "context window" grows, their attention on the middle section—the "muddle in the middle"—drops precipitously. This is exactly why my Chapter 1 was perfect, but the core of my book became a work of fiction.

3. Context Drift (The Fork in the Tree)

LLMs are Next-Token Prediction engines. They don't "know" facts; they predict the most likely next word based on the patterns they see. Context Drift occurs when a small, confident error is made early on. Because the model treats its own previous output as "truth," it builds the next sentence on that error.

Like a branch forking off a tree, once the AI drifts, it never returns to the trunk of your reality. It is now drafting a better version of its own hallucination.

The "Glove" Reality: Hallucination as a Feature

The most controversial realization for me was this: Hallucinations are not a bug. OpenAI CEO Sam Altman has described hallucinations as part of the "magic" of generative AI. If you forced a model to only say things it was 100\% certain about, it would lose the "creative" reasoning that makes it useful. It would be a 1990s database, not a 2026 thought partner. Andrej Karpathy, a founding member of OpenAI, famously described LLMs as "dream machines"—hallucination is their primary feature; it’s just that most of the time, they are dreaming within the bounds of our facts.

Expecting an AI not to hallucinate is like complaining that wood gives you splinters. The wood isn't the problem—that is the nature of the material. The problem is that the professional wasn't wearing gloves.

When a judge sanctions a lawyer for a fake citation, they are punishing someone for not understanding the physics of their own tools. We cannot stop the wood from being wood. We can only change how we handle it.

In the legal profession, we are trained to delegate. We hand tasks to junior associates, paralegals, and law clerks with the expectation that they will follow our instructions. But there is a fundamental difference between delegating to a human and delegating to an LLM.

When you delegate to a human, you are delegating to an agent with a shared reality. If a junior associate finds a gap in the case law, they stop and ask for clarification. When you "delegate" to an AI, you are handing a task to a prediction engine. It does not have a shared reality; it has a statistical probability map.

As the Model Rules of Professional Conduct (specifically Rule 1.1 regarding Competence) continue to evolve in the face of generative technology, the "Physics" described above becomes a matter of ethics. If you do not understand Sycophancy or Context Drift, you cannot effectively supervise your "non-human assistant." You aren't just using a tool; you are managing a masquerade. The duty of "Technological Competence" now requires us to look past the beautiful prose and interrogate the underlying architecture.