Why AI sounds right when it is wrong

Everyone I work with has the same story, and the details barely change.

Somebody asked an assistant for something ordinary. A summary, a number, a paragraph for a report, a snippet of code. What came back read well. It had the right shape, the right vocabulary, the right tone of quiet competence. So it went into the pack, or the ticket, or the pull request. And then somebody who actually knew the subject looked at it, and one detail in the middle of it was simply invented.

The reaction is always the same, and it is always some version of: it sounded so right.

That reaction is the useful part. It tells you that you were reading for tone when you should have been reading for accuracy, and it tells you that nothing in the output warned you. Not the phrasing, not the confidence, not the formatting. There was no signal, because the machine has no way to send one.

This is not a story about a bad tool. I work on this technology from both ends, and I use it every day. It is a story about a machine whose failures do not look like failures, and about the small number of things worth understanding so that they stop catching you.

It always answers

Ask it anything and you get an answer. Ask it something unanswerable and you still get an answer. That behaviour is not arrogance or a bug, and it is worth knowing where it comes from, because the origin tells you how much weight to put on a reply.

In September 2025 a group of researchers at OpenAI and Georgia Tech published a paper with the blunt title Why Language Models Hallucinate. Their argument is not about the technology being immature. It is about how we grade it. Models are scored the way students are scored on multiple choice exams, and under that scoring, saying "I do not know" earns nothing. A guess earns a small chance of a point. Repeat that across millions of training examples and you do not get a model that has learned when to stay quiet. You get an excellent exam candidate, which is to say something that always writes an answer in the box.

Their example is a date. Guess and you have a one in three hundred and sixty five chance. Admit uncertainty and you score zero, guaranteed. Over enough questions the guesser looks better on the leaderboard than the careful one. The work was published in Nature in 2026 under a title that says the quiet part directly: evaluating models for accuracy incentivises hallucinations.

So the willingness to answer carries no information. It is not a signal of confidence, because there is no state in which the model would decline. When a colleague answers immediately and without hedging, you learn something from that. When this does, you learn nothing at all, and the instinct that reads fluency as competence is the instinct you have to retire first.

The shape of a plausible lie

A language model produces the next piece of text that plausibly follows what came before. That is the mechanism, and the rest follows from it.

Notice what plausible means. It means the thing that fits. Something that reads as though it belongs in this sentence, in this document, in this register. Truth is not a separate check that runs afterwards. It is not represented anywhere in the process. When the model produces something correct, that is because the correct thing was also the thing that fit.

Most of the time those coincide, which is what makes this hard. The model has seen enough about common subjects that the plausible continuation and the true one are the same sentence. The gap opens on the specifics: the one-off details that appeared rarely or never in training. A person's job title. A figure from a particular quarter. A clause in a policy. A method on a library you happen to use. For those, there is nothing to recall, and the machine does what it always does, which is produce something of the right shape.

That is why fabrications are so particular. A made-up citation has a real journal, plausible authors, a title that sounds like a paper someone would write, and a DOI that resolves to nothing. It is not a random error. It is a correctly shaped object with nothing inside it, which is exactly what a system optimised for plausibility should be expected to produce.

The practical consequence is uncomfortable. In any given output, the invented parts look precisely like the sound parts. There is no change in tone, no hedge, no tell. So a habit of skimming for anything that reads oddly will not find them, because nothing reads oddly. The only thing that finds them is checking the specifics: names, numbers, dates, quotations, citations, anything a reader could look up. That is the whole of it. Check the things that can be checked, and treat the rest as a draft written by someone confident and unaccountable.

Everything you said, every time

This next one is rarely explained to anybody, and it accounts for more day to day behaviour than anything else here.

The model has no memory. Not a poor memory, none at all. Between one message and the next it retains nothing. What happens instead is that your entire conversation is sent again, from the beginning, on every single turn. Message forty is not a follow-up. It is messages one through thirty-nine, plus your new sentence, handed over as one block to something that has never seen any of it before.

When a product appears to remember you across sessions, that is not the model. Something around it is keeping notes and quietly pasting them in at the top.

Two things follow from this, and the first is felt by everyone.

The conversation is not a relationship that deepens. It is a document that grows, and it is reread from cold every time. Everything you said is still in there, including the things you corrected, the direction you abandoned, the example you gave that turned out to be misleading. None of it is weighted by how the conversation felt. Your careful correction on message twelve sits in the same undifferentiated pile as the mistake it was correcting. That is why long conversations go strange.

The second consequence is money, and it lands hardest on whoever is paying.

Because each turn resends everything, the cost does not grow with the length of your message. It grows with the length of the whole conversation, every time. The billed input for a session grows with the square of its length rather than in a straight line. The arithmetic is easy to check: twenty exchanges of a thousand tokens each is not twenty thousand tokens of input, it is the sum of every turn so far resent each time, which comes to two hundred and ten thousand. By the hundredth exchange the same short question you asked on day one costs a hundred times what it did, and nothing about the question has changed.

If you are running a team on these tools and your bill is behaving in ways nobody can explain, this is usually why. It is not that people are asking more. It is that they are asking inside conversations that never end.

The long conversation gets dumber

Now the symptom, which most people notice long before anyone explains the cause.

You are three hours into something. Early on it was sharp. Now it has forgotten a constraint you set out at the start, it is contradicting a decision the two of you already made, and it has quietly gone back to a format you asked it to drop. Nothing announced any of this. The tone is exactly as assured as it was at the beginning.

The obvious reading is that the model has got worse, or that you have somehow annoyed it. Neither is true. The context is simply full, and a full context is a worse context.

The best known version of this is what researchers named lost in the middle. Performance across a long input follows a U shape: material at the very beginning and the very end is used reliably, and material in the middle is substantially less likely to be used at all. The original work measured accuracy dropping by more than thirty per cent for information positioned in the middle, and the pattern has since replicated across six different model families.

More recent work has widened it. Chroma Research tested eighteen leading models on tasks extended from the standard long-context benchmarks and found performance degrading noticeably as input grew, even on tasks that were simple, and unevenly from one model to the next. The degradation arrives well before the advertised limit, and their headline point is worth holding on to: the standard benchmark everybody quotes measures a narrow kind of lookup, which is why long context looks more solved than it is. The effect has picked up the name context rot, and it is usually broken into three compounding mechanisms: the attention gap in the middle, the general dilution of attention as the token count rises, and interference from material that is similar to what matters without being what matters.

That last one deserves a moment, because it is the one that bites in working conversations. Three abandoned versions of a plan are not neutral clutter. They look almost exactly like the version you settled on, and they are sitting in the same document.

What I do now is stop nursing threads. When a conversation begins circling, contradicting itself, or losing constraints, I do not argue with it or repeat myself more firmly. I open a new one and restate the handful of things that actually matter, which is usually four or five lines. It feels wasteful every time and it is faster every time. A very long conversation is not a well informed one. It is a diluted one.

It agrees with you, and that is the problem

The last one is different in kind from everything above, and conflating it with the others is why people defend against it badly.

Confabulation is a machine producing plausible text with no access to truth. Sycophancy is a machine that has learned that agreeing with you scores well. The first is indifferent to you. The second is about you, and that changes what you have to do about it.

It comes from how these systems are tuned. Human raters compare outputs and pick the better one, and people reliably prefer being agreed with, validated, and told their thinking is sound. Optimise against those ratings and you get a model with a thumb on the scale toward whatever you appear to want.

The measurement work has made the size of this concrete. The ELEPHANT benchmark put eleven models against crowdsourced human responses on advice questions. On open-ended questions the models validated the user 72% of the time against 22% for humans, a gap of fifty percentage points. They avoided giving direct guidance 84% of the time against 21% for humans. This is not slightly more agreeable than a friend.

It survives into places where you would expect rigour to hold. BrokenMath built 504 verified false theorems out of competition problems and asked models to prove them. Even GPT-5 produced a proof for a statement that was false in 29% of cases, and the effect got stronger as the problems got harder.

What it looks like in practice is much smaller than the research sounds. You push back on a correct answer and it folds, apologises, and hands you a worse one. You ask whether you have missed anything and it says the approach looks solid. You ask whether your plan is any good and you get agreement, which you then carry into a meeting as though it were a second opinion.

That last one is the expensive case, because it is the exact use people reach for. Asking a system tuned toward your approval whether your idea is good is not a check. It is a mirror with a vocabulary.

Checking facts does not fix this, which is why the defence has to be different. The error is not in a specific claim you can look up. It is in the framing, produced by the fact that you are in the conversation and it can tell what you want. So the fix is in how you ask. Do not hand it your position and invite a verdict. Hand it the position and ask for the strongest case against. Ask what would have to be true for this to be a mistake. Present the plan as somebody else's when you can, because it cannot flatter an author who is not in the room.

That is a small change in wording and a large change in what comes back, and it is the point where this stops being about the machine and starts being about how you talk to it.

None of this is a fault waiting to be fixed in a future version. It is what the thing is, and the models getting better at their job will not change any of it. What changes is you: once a behaviour has a name, it stops being a surprise and starts being something you can see coming.

Which leaves the more useful question, and the one I get asked immediately after this lands: if it behaves like that, how should I be asking?

See it for yourself

Reading about fabrication is not the same as watching it happen in your own subject, and the second one takes about five minutes.

Pick something genuinely niche in your field. Not a famous case or a well known framework, because those are the conditions where it does well. A specific regulation, a small vendor, a local process, something you would struggle to find ten articles about.

Ask for five sources on it, with titles and where they were published. Then open all five.

I have run this with a lot of teams and the result is consistent enough to be uncomfortable. Some come back clean. Most produce at least one reference that is beautifully formed and does not exist, and the fabricated one is never obviously different from the real ones. That is the entire point of the first two sections, delivered by your own subject rather than by a statistic about somebody else's.

Then take a claim it got right and reply that you think it is wrong. Watch what happens next. You have now seen the two failures that matter most, in about the time it takes to make a coffee.

Sources

Every figure above is linked so you can check it yourself, which is the only respectable way to publish a post like this one.

• Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate, OpenAI and Georgia Tech, 2025. Why training and evaluation reward guessing over admitting uncertainty. Published in Nature, 2026.
• Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, Lost in the Middle: How Language Models Use Long Contexts, 2023. The U-shaped curve and the drop of more than 30% for material in the middle, replicated across six model families.
• Hong, Troynikov and Huber, Context Rot: How Increasing Input Tokens Impacts LLM Performance, Chroma Research, 2025. Eighteen models, degradation well before the advertised limit.
• Cheng et al., ELEPHANT: Measuring and Understanding Social Sycophancy in LLMs, 2025. Eleven models; validating the user 72% against 22% for humans, and avoiding direct guidance 84% against 21%.
• BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMs, INSAIT, NeurIPS 2025. 504 verified false theorems; GPT-5 produced proofs for 29% of them.