How many tokens per second is good?
A token is a piece of a word: in English, about three quarters of a word on average, so 100 tokens is about 75 words. Most people read 200 to 300 words a minute, which is 5 to 8 tokens a second. From that:
- Under 5 tokens a second is slower than you read: fine for a short answer, a wait for a long one.
- 10 to 20 is comfortable: the answer appears faster than you read it.
- 30 and up matters for code, long documents and reasoning models, which write a long stretch of thinking before they answer: 2,000 tokens of thinking takes 100 seconds at 20 tokens a second, and 20 seconds at 100.
- Above 60 or so, more speed is hard to see when you read along, but it still shortens the wait for long answers and for apps that run a model many times over.
Two speeds: reading the prompt, and writing
Before it writes anything, a model reads what you sent it — your question, the conversation so far and any document. That is measured in tokens per second too, and it is far faster: hundreds to thousands a second on a graphics card, so a short question is read in a blink. It matters for long documents, and it is where a graphics card is furthest ahead of a processor. The writing speed is the one people mean by “tokens per second”, and the one shown above.
What sets the speed
To write each token, a model reads all the weights it uses from memory. So the speed is roughly how fast that memory can be read, divided by the size of the model:
- A graphics card’s memory is read at 200 to 1,800 GB a second, depending on the card — which is why a model that fits on the card is fast.
- System memory is read at 50 to 100 GB a second in a desktop with two channels, so a model on the processor is several times slower.
- A smaller file of the same model writes faster: a 4-bit file is read nearly twice as fast as an 8-bit one. Which GGUF file to download explains the sizes.
- Mixture-of-experts models, such as gpt-oss and Qwen3 30B-A3B, use only a part of themselves for each token, so they write as fast as a much smaller model once they fit.
Tokens per second by graphics card
Best GPU for local AI lists the estimated speed of four popular models on every graphics card, and each card’s page lists every model it runs. Can my PC run AI? works it out for your whole PC: graphics, memory and drive.
Slower than it should be?
If a model writes far more slowly than its card should manage, part of it is usually on the processor rather than the card. Ollama using the CPU instead of the GPU and LM Studio not using the GPU show how to tell and fix it, and Bottleneck’s free app checks it in one click.
See how fast your model really runs.
Bottleneck’s free app checks a model run with Ollama, LM Studio, llama.cpp, KoboldCpp or Jan: where it really runs, how many tokens a second it writes, and what your PC should manage with it. Nothing leaves your PC.