Lesson 10 of 28 · Language models and chat assistants

What is a context window?

A context window is the maximum amount of text, measured in tokens, that a language model can consider at one time. It holds your instructions, the conversation so far, any files you attach and the model's reply. Text outside the window is invisible to the model, which is why very long chats can lose track of earlier details.

Think of it as the model's desk

Imagine a desk with room for only so many pages. The model can read anything on the desk, but nothing that is not on it. When the desk is full, something has to come off before anything new can go on. A context window works the same way, except that the pages are tokens.

Crucially, the model has no other memory of your chat. Every time you send a message, the app puts the whole conversation back on the desk along with your new message, and the model reads it all again from the start.

What fills the context window

Everything in a request shares one budget:

The system prompt: hidden instructions that set the assistant's role and rules.
The conversation history: your earlier messages and the model's earlier replies.
Attached material: pasted text, uploaded files, search results and tool outputs.
The reply itself: the tokens the model writes also count towards the limit.

What happens when it fills up

When a conversation or document no longer fits, the app has to choose: drop the oldest messages, replace them with a short summary, refuse the request or ask you to start a new chat. That is why a long chat can "forget" an instruction you gave at the start.

Context windows have grown considerably over time, making it possible to process whole reports or codebases in a single request. But a large window is not the same as perfect attention: the 2023 study "Lost in the Middle" (Liu et al.) found that models can miss information placed in the middle of long inputs.

What a context window is not

It is not long-term memory: once a chat ends, its contents are gone from the model's view. Some apps offer a memory feature that saves notes and slips them into future chats, but that is the app at work, not the model learning. Nor is it training: pasting a document in does not teach the model anything permanently.

And bigger is not automatically better. With API pricing, every token in the window is billed as input on each request, so stuffing in more context costs more and can slow replies down. Retrieval-augmented generation (RAG), which fetches only the relevant passages, is often cheaper than sending everything.

Why it matters when choosing a tool

If you work with long contracts, research papers, transcripts or large codebases, check how much context a product supports and how it handles overflow. For short everyday tasks, almost any modern assistant has enough room. Mindber's LLM cost calculator shows how input size drives cost, and the Arena leaderboard helps you compare models.

Example

Budgeting a context window

Worked example

Context window: 8,000 tokens (kept small for illustration) System prompt 500 Conversation so far 5,000 Pasted document 2,000 ───────────────────────────── Already used 7,500 Left for the reply 500 → the answer may be cut short, or the app drops the oldest messages to make room

Real context windows are usually far larger, but the arithmetic is the same: instructions, history, attachments and the reply all draw on one budget.

Key points

The context window is how many tokens a model can consider at once.
Instructions, chat history, files and the reply all share the same window.
Anything outside the window is invisible to the model.
Bigger windows allow longer documents but cost more per request.
Models can still miss details in the middle of very long inputs.

Try it on Mindber

See how context size drives cost

AI model leaderboard (Arena)

Glossary: retrieval-augmented generation (RAG)

Previous lesson: What is a token in AI?

Frequently asked questions

Does a bigger context window make a model smarter?

No. A bigger window lets the model see more text at once, but it does not improve its reasoning, and research such as the "Lost in the Middle" study (Liu et al., 2023) found that models can overlook information in the middle of long inputs. Clear, focused context often beats simply adding more.

Why does an AI chatbot forget what I said earlier?

Usually because the conversation has grown longer than the context window, so the app dropped or summarised the oldest messages. Starting a new chat with a short summary of the key facts, or repeating important instructions, helps keep them in view.

Is the context window the same as memory?

Not exactly. The context window is temporary and holds only the current request. Some apps add a memory feature that saves notes about you and inserts them into the window in later chats, but the underlying model itself does not remember past conversations.