Glossary
Prefill and decode
The two phases of inference: reading the whole prompt in one parallel pass (prefill), then generating the answer one token at a time (decode).
Prefill is fast per token because every input token can be processed at once; decode is sequential and memory-bound, which is why output tokens cost roughly five times input tokens on every major price list. Serving systems batch many users' decode steps together to keep accelerators busy.
Related terms
Active parametersAPIAttentionCost per taskGPU
Get the brief in your inbox
One email every morning: the stories that mattered in AI, each with a one-line "why it matters" for your role. Free, no ads, unsubscribe any time.
Published daily by AI Briefing. We never share your address.