What are parameters in an LLM?

A large language model (LLM) is a program that reads text and predicts what comes next, one token at a time (a token is a word or a piece of a word). Inside it is a neural network: layers of simple maths operations connected to each other. Each connection has a strength, stored as a number. Those numbers are the parameters, also called weights.

Picture a giant sound-mixing desk with billions of tiny knobs. No single knob means anything on its own, but together their positions decide what comes out of the speakers. In an LLM, the knob positions decide which next token looks most likely. Grammar, facts, writing styles and coding habits are all stored as patterns spread across those settings, not as entries in a list.

Before training, the parameters are set to random values, so the model produces gibberish. Training shows it enormous amounts of text, one passage at a time. For each passage the model guesses the next token, compares its guess with the real one, and every parameter is nudged slightly in the direction that would have made the right answer more likely. Repeat this billions of times and useful patterns emerge.

Once training ends, the parameters are frozen. When you chat with a model, it uses its parameters but does not change them, which is why a model does not permanently learn from your conversation. Changing them again requires more training, such as fine-tuning.

Not automatically. Research on scaling (Kaplan et al., 2020) found that language models tend to improve predictably as parameters, training data and computing power grow together. Later work (Hoffmann et al., 2022) showed that many large models had been trained on too little data for their size, and that a smaller model trained on more data could outperform a bigger one.

Size also has costs. Every parameter must sit in memory while the model runs, so larger models need more powerful hardware and are usually slower and more expensive per answer. Some designs, called mixture-of-experts, use only part of their parameters for each token, so you may see both a 'total' and an 'active' parameter count.

Parameters are not the settings you can change in an app. Options such as temperature or maximum length are sometimes called 'parameters' in API documentation, but they only steer how the model produces an answer; they do not touch the learned weights. Parameters are also not a copy of the training data, and the count says nothing about the context window, which is how much text the model can read at once.

You will meet parameter counts in model names (such as '8B' or '70B'), on model cards and download pages for open-weight models, and in model comparisons. Many companies do not disclose the size of their closed models, so a missing number is common. For a sense of scale, the 2020 GPT-3 paper described a model with 175 billion parameters (Brown et al., 2020).