Inference

Inference is the stage where a trained AI model is used to produce output: it takes an input, such as a prompt or an image, and generates a prediction or response. Training builds the model once; inference happens every time someone uses it, so it drives speed and running cost.

During inference, the model's weights are fixed. For a language model, inference means processing the input tokens and then generating output tokens one at a time until the answer is complete. It runs on GPUs or specialised chips in the provider's data centre, or on local hardware for smaller and open-weight models.

Why it matters: when you pay for an AI API or subscription, you are mostly paying for inference. Latency, rate limits and per-token prices all reflect inference costs, and techniques such as batching, caching and smaller models exist to reduce them. For privacy-sensitive work, where inference runs, whether in the vendor's cloud, a specific region or on your own devices, can matter as much as the model itself.

Example: each time an assistant answers a question, the provider runs one inference request and bills the input and output tokens.

Note: inference (running a model) is not the same as reasoning (a model working through a problem step by step), although reasoning models spend extra inference compute thinking before they answer, which raises cost and latency.