Token cost is an architecture decision

Cost, in ordinary backend work, is something you deal with afterwards. You ship the feature, you watch the graphs, and when the bill looks wrong you go and find the query that turned out to be running on every page load. It is an operations problem with an operations fix, and the design rarely has to change.

That reflex does not survive contact with this work, and I have watched it fail the same way more than once. A team builds something careful and well tested that does exactly what was asked. It goes in front of real users, the invoice arrives, and nobody in the room can name the decision that caused it, because it was made weeks earlier and did not look like a cost decision at the time. It looked like an architecture decision. It was both.

The meter here runs on every token in and out, which puts price inside the design loop rather than after it. Nothing in ordinary backend engineering trains that instinct.

A warning before the detail. Every price below is dated and every one will eventually be wrong. The mechanisms will not be.

The meter counts what you send, not just what you get back

Start with the unit. Providers do not bill per request or per conversation. They bill per token, which is roughly a chunk of a word, and they bill in two separate columns: the tokens you send in, and the tokens that come back.

Everything you hand over counts as input. The system prompt and the tool definitions. The documents you retrieved and the conversation so far. All of it is metered on every single call, and for most real features the thing you send is far larger than the thing you get back.

If you read the Layer 1 post on why AI sounds right when it is wrong, you already have the mechanism underneath this. The model holds nothing between turns, so your whole conversation is resent from the beginning every time, and the billed input for a session grows with the square of its length rather than in a straight line. That post made the point for one person in one chat window. This is the same mechanism with a product in front of it.

Now the part that catches engineers out. The gap between two reasonable ways of building the same feature is not a few per cent.

Anthropic, writing up their own multi-agent research system, report that "agents typically use about 4x more tokens than chat interactions, and multi-agent systems use about 15x more tokens than chats". Take that for exactly what it is: an internal measurement with no published methodology, from a company that sells tokens. An order of magnitude, not a figure. Their conclusion is the sentence worth keeping, that "multi-agent systems require tasks where the value of the task is high enough to pay for the increased performance".

Think about what that does to a roadmap. "Summarise the customer's recent activity" is one line on a board. It can be a single call with the activity pasted in. It can be an agent that searches, reads, decides it needs more, and searches again. It can be a supervisor handing work to three sub-agents and reconciling what comes back. Same line on the board, same demo, and a running cost that differs by roughly the order of magnitude above. The architecture picks the price band, and it picks it before the first user arrives.

Cost and latency move together here, because the tokens that get billed also get processed. A long prompt is a slow prompt. I am not going to give you a number for that. Every production latency benchmark I could find traced back to a vendor page or a content farm, several citing each other, and not one published a method.

Caching is prefix-based, so the order you assemble a prompt in is architecture

Prompt caching is the discount that changes how you write code. If the beginning of your prompt is identical to the beginning of one you sent recently, the provider does not have to process it again, and charges you a fraction of the usual rate for that stretch.

The word that matters is beginning. The match runs from the first character forward and it stops at the first difference. Anthropic's documentation puts the requirement plainly: "Cache hits require 100% identical prompt segments, including all text and images up to and including the block marked with cache control." Not similar. Identical.

The published terms, as of September 2026, all taken from the providers' own documentation:

• Anthropic: a five-minute cache write costs 1.25 times the base input price, a one-hour write 2 times, and a cache read 0.1 times, dropping to 0.025 times on their newest models.
• OpenAI: reused tokens are billed at a reduced cached-input rate, "discounted up to 90%", with a default minimum cache lifetime of thirty minutes and up to twenty-four hours on some models. Their caveat matters as much as the discount: "Cached states live on individual machines, where traffic above 15 requests per minute can lead to overflow routing." A cache hit is not something you get to assume.
• Google: cached input on current Gemini models is billed at roughly half the standard input rate, and cache storage is charged separately by the hour, in the region of $0.50 to $1.80 per million tokens per hour.

Those are published commercial terms rather than anybody's benchmark, which makes them unusually reliable and completely perishable. I checked all three against notes from a few weeks earlier and one had already moved, which is the best argument for checking them yourself.

Then the consequence. Because the match is a prefix, the order in which you assemble a prompt decides whether you get the discount at all. Stable content first: system instructions, tool definitions, examples, whatever context does not change between calls. Volatile content last: the retrieved chunk, the current state, the user's message.

Put a timestamp at the top of your system prompt and you have quietly switched the discount off for every call your service makes. Nothing will tell you. There is no error and no warning. The output is correct. The line item is simply several times larger than it needed to be, and it stays that way until somebody goes looking.

I like this corroboration, because it arrives from an unexpected direction. The Model Context Protocol specification, revision 2026-07-28, now says servers "SHOULD return tools from tools/list in a deterministic order to enable client-side caching and improve LLM prompt cache hit rates", and requires new ttlMs and cacheScope fields on list results through a CacheableResult interface. A wire protocol has changed its rules to protect somebody else's cache hit rate. Micro-optimisations do not get that treatment.

Nobody has measured how many teams actually do this, so treat any uptake figure you are shown with suspicion.

Not everything has to answer now

Batching is the second discount and it is almost embarrassingly easy to take.

Rather than sending a request and waiting for the answer, you hand the provider a collection of requests and collect the results later. As of September 2026, both major providers charge half price for it. OpenAI's Batch API gives a "50% cost discount compared to synchronous APIs" over a twenty-four hour completion window, and, usefully, "using the Batch API will not consume tokens from your standard per-model rate limits". Anthropic's Message Batches API also halves the cost on both input and output, with most batches finishing in under an hour and a hard twenty-four hour expiry. It stacks with prompt caching, although cache hits inside a batch are best effort.

Half the bill, in exchange for not having the answer this second.

So the design question is which parts of your feature genuinely need to answer now, and the honest count is lower than most people assume. Classifying yesterday's support tickets. Back-filling a column across a table. Running your eval suite. Scoring a queue that a human works through the next morning. None of that is interactive. Most of it ends up synchronous because the first version was written that way and nobody went back to it.

In a busy system the separate rate limit pool is worth as much as the discount, because bulk work stops competing with live traffic.

Gateways, routing and running more than one model

Running more than one model is now the ordinary case. a16z's Q1 2026 enterprise survey puts 81% of enterprises on three or more model families in test or production, up from 68%.

Read that number alongside its own disclosure, because a16z did something rare and stated the limits themselves. Participants were "selected from [our] proprietary database using targeted screening... not randomly sampled", and "the findings may not be representative of the broader market", and, separately, "we are investors in OpenAI." The sample is a hundred senior people at large enterprises. A survey that tells you why it might be wrong is worth more than one that does not, and very few in this field do.

Take the direction rather than the decimal. That plurality is what the gateway slot exists to serve. LiteLLM, OpenRouter, Portkey and Cloudflare AI Gateway are the current occupants. The slot will outlive all of them, and what it does is the durable part: one API across providers, automatic fallback when a provider is degraded or rate-limiting you, cost attribution by team or feature or customer, and a spend ceiling enforced in one place rather than hopefully in every service.

Now a caution about the idea here that sounds best and behaves worst. Semantic caching means matching questions that mean roughly the same thing, rather than exact prefixes, and serving the earlier answer. Genuinely appealing, and the tooling is not in good health. GPTCache, the most cited open-source implementation, has had no commit since July 2025. It carries over eight thousand GitHub stars, which is the clearest demonstration I know of why stars are a weak signal. Its own README discloses the failure mode: "In a semantic cache, you may encounter false positives during cache hits and false negatives during cache misses."

In operational terms, a false positive on a cache hit means somebody gets the answer to a question they did not ask.

Chip Huyen reached the same place in 2024, calling the value of a semantic cache "more dubious because many of its components are prone to failure", and noting that setting the similarity threshold "can also be tricky and require a lot of trial and error".

Prefix caching saves money with no correctness risk, because the match is exact. Semantic caching buys the saving with a correctness risk you have to price. Learn the failure mode before the discount.

Self-hosting, and the fact that your model has an end of life

Sooner or later somebody asks whether you should run the weights yourself. vLLM and SGLang are the two active serving standards, Ollama covers local and development work, and llama.cpp covers edge, CPU and quantised models.

The instructive entry is the one that is gone. Hugging Face's Text Generation Inference, the obvious choice a couple of years ago, is an archived repository now, and their own text points elsewhere: they "contribute to and recommend using going forward: vllm, SGLang, as well as local engines... like llama.cpp or MLX." Infrastructure in this space does get formally retired, and the notice is often a paragraph in a README.

How common is self-hosting? The evidence is thin and I would rather say so than fill the gap. The best methodology-disclosed source I found is Menlo Ventures' 2025 enterprise survey, 495 US decision-makers fielded that November by an independent firm, which put open-weight models at 11% of enterprise API market share, down from 19% the year before. Read the scope carefully. That measures which models organisations choose, not whether they run the infrastructure themselves. Different questions, and nobody appears to have measured the second properly.

Simon Willison, closing out 2025: "I have yet to try a local model that handles Bash tool calls reliably enough for me to trust that model to operate a coding agent on my device." One practitioner's view, worth exactly that.

Which brings the category to the thing sitting underneath all of it. Your core dependency is deprecated on a published schedule, changes behaviour underneath you, and reprices when it moves.

The first post in this series had the table. OpenAI announced the Assistants API's deprecation in August 2025 with shutdown a year later, and Agent Builder's in June 2026 with shutdown that November, roughly fourteen months from launch. That is the platform layer, and the model layer runs the same treadmill more often.

Tian Pan's framing is the most useful sentence I have read on this: stop treating model identifiers like brand names and start treating them like library versions with a published end of life. What follows is ordinary engineering discipline applied somewhere new. Pin snapshots rather than aliases. Put the model version in your logs and your traces, so you can answer "which model produced this" months afterwards. Put deprecation dates in a calendar. Run your eval suite against the successor before you are forced to move. And give the migration a named owner, because "migrations that are everyone's job are nobody's".

His list of what breaks is worth reading twice: refusal behaviour your prompts were tuned around, JSON parsers failing on output format drift, a router quietly falling back to a deprecated model that no eval covers, and cost baselines. That last one is on the list for the reason this post exists. A migration you did not plan for is also a repricing you did not plan for.

Put the shutdown date of the model you depend on into the same calendar as your certificate renewals. It is the same kind of date, and it will arrive with the same amount of warning.

See it for yourself

All of this is checkable against your own bill in about half an hour, and the first number is usually the surprise.

Open your provider console for last month and split the spend into input and output. Most teams find input dominates by a wide margin, which is the opposite of how they had been thinking about it, because output is the part you read.

Now find your cache read figures. If your console reports cached tokens as a share of input and that share is close to zero, you are paying full price for a prefix you send on every single call.

Then take your highest-spend endpoint and dump the exact prompt string it sends on two consecutive calls to a file. Diff them. You are looking for the first character that differs, and how far down it sits. Everything after that point is uncached, by definition.

Finally, list your ten highest-volume jobs and mark the ones that genuinely need an answer within a second. The ones you cannot honestly tick are the ones you have been paying double for.

Sources

Linked so you can check them, and dated because half of them are commercial terms that move.

• Anthropic, Prompt caching, checked 1 September 2026. Vendor pricing documentation. Verbatim: "Cache hits require 100% identical prompt segments, including all text and images up to and including the block marked with cache control." Five-minute cache write 1.25x base input price, one-hour write 2x, cache read 0.1x, and 0.025x on their newest models. Live commercial terms: re-check before relying on the multipliers.
• Anthropic, Batch processing, checked 1 September 2026. Vendor pricing documentation. 50% cost reduction, most batches finishing in under an hour, results expiring at twenty-four hours, and caching discounts that stack with batch on a best-effort basis for cache hits.
• OpenAI, Prompt caching, checked 1 September 2026. Vendor pricing documentation. Cached input "discounted up to 90%", a thirty-minute default minimum cache lifetime with longer retention on some models, and the caveat "Cached states live on individual machines, where traffic above 15 requests per minute can lead to overflow routing."
• OpenAI, Batch API, checked 1 September 2026. Vendor pricing documentation. "50% cost discount compared to synchronous APIs", a twenty-four hour completion window, and "using the Batch API will not consume tokens from your standard per-model rate limits."
• Google, Gemini API pricing, checked 1 September 2026. Vendor pricing documentation. Cached input at roughly half the standard input rate on current models, with cache storage billed separately from $0.50 to $1.80 per million tokens per hour by tier. These figures had already changed from the ones I recorded a few weeks earlier, which is the clearest evidence in this post that vendor pricing is perishable.
• Anthropic, How we built our multi-agent research system, 13 June 2025. Source of "about 4x more tokens" and "about 15x more tokens", and of "multi-agent systems require tasks where the value of the task is high enough to pay for the increased performance." Internal measurement from a vendor, with no published methodology. An order-of-magnitude illustration and nothing more.
• Model Context Protocol, specification revision 2026-07-28 changelog. Verified at primary. Deterministic tools/list ordering "to enable client-side caching and improve LLM prompt cache hit rates", and the required ttlMs and cacheScope fields on the new CacheableResult interface. Cite the revision: this specification churns hard.
• a16z, Leaders, gainers, and unexpected winners in the enterprise AI arms race, Q1 2026. 81% of enterprises running three or more model families in test or production, up from 68%. A self-selected survey of 100 VP and C-level respondents at Global 2000 firms, and the quoted limitations and investor disclosure are the report's own words. Treat it as direction, not measurement.
• GPTCache, Zilliz. Over eight thousand stars, no commit since July 2025. README, verbatim: "In a semantic cache, you may encounter false positives during cache hits and false negatives during cache misses." The star count is a weak signal and is quoted here only to make that point.
• Chip Huyen, Building A Generative AI Platform, 25 July 2024. Semantic cache value "more dubious because many of its components are prone to failure", and thresholds that "can also be tricky and require a lot of trial and error." One named practitioner, no methodology, and consistent with what the tooling shows.
• Hugging Face, text-generation-inference. Archived, confirmed via the GitHub API. Their own repository text recommends "vllm, SGLang, as well as local engines... like llama.cpp or MLX" going forward.
• Menlo Ventures, 2025: The State of Generative AI in the Enterprise. 495 US enterprise decision-makers, fielded 7 to 25 November 2025 by an independent research firm. Open-weight models at 11% of enterprise API market share, down from 19%. Scope limit: this measures model and provider selection, not self-hosted versus hosted infrastructure. Do not read it as a self-hosting statistic.
• Simon Willison, The year in LLMs, 31 December 2025. The local-model tool-calling quotation. One practitioner's experience, offered as such.
• Tian Pan, The model deprecation treadmill, 27 April 2026. The library-versions-with-an-EOL framing, the prescription, and the list of what breaks. Opinion with no methodology. The engineer-week arithmetic in that post is one person's estimate and is deliberately not quoted here.
• OpenAI, deprecations. The Assistants API and Agent Builder dates. Published primary documentation, and the strongest single piece of evidence in this series that the platform layer moves underneath you.

Every price above is a live commercial term. If you are reading this a long way from September 2026, or republishing any of it, re-check all five vendor pages first.