What is inference, and why does it cost so much?

Inference is the act of running a trained model to get an answer. Training happens once, over weeks, on tens of thousands of chips. Inference happens every time someone sends a prompt, and because a popular model answers billions of requests a month, inference is where most of the electricity, most of the chips and most of the cost of AI now sit.

What happens when you send a prompt

A request has two phases. In prefill, the model reads your whole prompt at once and builds an internal representation of it, the key-value cache. This phase is parallel and fast per token. In decoding, the model produces the answer one token at a time, and each new token has to re-read everything that came before it. Decoding is sequential, which means it cannot be spread across chips as easily and ties up expensive memory for the whole time the answer is being written.

That asymmetry is why vendors charge far more for output than for input. In our pricing tracker the ratio is consistently about five to one: Claude Fable 5.2 is $8 per million input tokens and $40 per million output, GPT-6.1 Sol is $2 and $10, Meta's Muse Ember is $0.10 and $0.50. Output tokens are the scarce resource.

Where the money goes

Three things dominate. Memory bandwidth: a frontier model's weights are hundreds of gigabytes and must be streamed through the chip for every token, so the limit is how fast memory can feed the processor, not raw compute. Chips and power: an accelerator costs tens of thousands of dollars, draws a kilowatt or more, and is only useful while it is busy, so idle time is pure loss. Serving engineering: batching many users' requests together, keeping caches warm and routing traffic are what separate a provider that makes money at $0.50 per million output tokens from one that loses it.

Reasoning models add a fourth cost. When a model 'thinks' before answering, it is generating hidden output tokens, and you pay for them. A hard question at a high effort setting can burn ten times the tokens of the visible answer. This is why the price of a token and the price of a task have come apart: a model that is cheaper per token but talks more can cost more per completed job.

The levers vendors give you

Prompt caching stores the prefill of a repeated prefix so it does not have to be recomputed; cached input is typically a tenth to a fortieth of the normal input price. Batch processing lets the vendor schedule your requests when chips are idle, usually for a flat 50% discount, in exchange for results arriving within hours. Effort or reasoning budgets cap how many hidden tokens the model may spend. Fast modes do the opposite: pay double for priority access to faster hardware. Long-context surcharges apply when a prompt crosses a threshold, often 200K or 272K tokens, because very long prompts make the cache expensive to hold.

Why prices keep falling anyway

Every generation of hardware moves more memory bandwidth per dollar, every serving stack gets better at batching, and mixture-of-experts designs activate only a fraction of a model's parameters per token. The result is that the price of a given level of capability has fallen by roughly an order of magnitude every year or so. The catch is that usage grows faster. Agents that run thousands of small steps, reasoning modes that think for minutes, and context windows of a million tokens mean total inference spend keeps rising even as unit prices fall. The question for any budget is not what a token costs but how many tokens the job takes.