Model comparisons are full of numbers like “26 tokens per second.” Then you run the model on your own phone and it feels different from what the number implied. The number was probably accurate. It just wasn’t measuring the thing you actually experience.
Speed in language model inference is at least two different quantities, and the one that dominates how fast something feels is usually not the one being quoted.
This article sits under How LLMs Actually Run on Your iPhone.
The two numbers that matter
Time to first token (TTFT) is the delay between hitting send and the first word appearing. It covers tokenizing your input and running the whole prompt through the network — the prefill phase.
Tokens per second (TPS) is how fast words appear once generation has started.
These are independent, and they respond to different things. TTFT scales with how long your prompt is. TPS scales with model size and memory bandwidth. A long prompt with a small model gives slow TTFT and fast TPS. A short prompt with a big model gives fast TTFT and slow TPS.
Perceptually, TTFT usually wins. A pause before anything happens reads as the app being stuck. Once text is flowing, people read along and stop noticing the rate — up to a point.
The thresholds worth knowing
Comfortable adult reading is roughly 240 words per minute, which is about 4 words per second, or 5 to 6 tokens per second. That sets the floor.
| Tokens/sec | How it feels |
|---|---|
| 5–8 | Slow. You are waiting on the model. |
| 10–15 | Fine. Text arrives as fast as you read. |
| 20–40 | Immediate. Text outruns reading. |
| 50+ | No perceptible improvement over 40. |
The important consequence: past about 25 tokens per second, extra speed buys almost nothing. A model at 45 tokens per second is not “better” than one at 26 in any way you can feel — both are already faster than you can read.
This is why raw speed is a bad reason to pick a smaller model. If a 2B model at 45 TPS and a 4B model at 25 TPS both outrun your reading, the only difference you will actually notice is answer quality. Pick on quality.
Where speed genuinely matters is at the low end — a 9B model on a device that can barely hold it, producing 6 tokens per second, is a real usability problem regardless of how good its answers are.
Why your numbers won’t match the benchmark
Published figures are measured under near-ideal conditions. Daily use is not near-ideal. Four things account for most of the gap.
Thermal state. This is the big one. A phone that has been idle runs at full clocks. After several minutes of sustained generation, iOS reduces clock speeds to control heat. Expect 25–40% off peak on a warm device. It recovers on its own once the phone cools.
Conversation length. The key-value cache grows with the conversation, and every new token must read more of it. Late in a long chat, generation is measurably slower than at the start — see Context Windows Explained for the mechanism.
Memory pressure. Other apps compete for the same unified memory and the same bandwidth. A phone with many apps resident produces lower throughput than a freshly-restarted one.
Battery state. Low Power Mode reduces peak performance deliberately. Below roughly 20% battery, throughput drops noticeably.
A 30–40% gap between a quoted benchmark and your Tuesday afternoon is normal and does not indicate anything wrong.
Why smaller models are faster, precisely
The reason is more specific than “less work to do.”
Generating one token requires reading every weight in the model from memory. Not a subset — all of them, every token. Modern chips do arithmetic far faster than they can move bytes, so throughput is set by memory bandwidth, not compute.
That gives a clean approximation:
tokens per second ≈ memory bandwidth ÷ model size in bytes
Halve the model’s size and you roughly double throughput. This predicts real behaviour well — and it explains why quantization speeds models up as well as shrinking them. A 4-bit model reads a quarter of the bytes its 16-bit version would.
It also explains the spread across Cloaked’s catalogue on one device: on an iPhone 15 Pro, Qwen 3 0.6B runs at roughly 45 tokens per second while Qwen 3.5 9B manages about 10. The ratio of sizes predicts the ratio of speeds more accurately than any benchmark suite.
Where local inference actually wins
On raw throughput, a data centre GPU beats a phone and always will. But throughput is only part of what you experience.
No network round trip. A cloud request travels to a server, waits in a queue, and travels back. That is typically 200–800ms before generation even begins, and far worse on a weak connection. A local model has none of it.
No queue, no rate limit, no capacity. Local speed doesn’t degrade at peak hours. There is no “high demand” message, no throttling, no monthly cap.
Consistent behaviour offline. On a plane, a train, or a basement, a local model runs at full speed while a cloud assistant produces nothing at all. How to Use AI Without Internet Access covers that scenario directly.
The honest summary: cloud models finish long answers sooner. Local models start sooner and never fail to start. For short and medium exchanges — the overwhelming majority of real use — the local experience is competitive, and on a bad connection it is simply better.
Practical speed tuning
If a model feels slow, in order of effectiveness:
- Start a new conversation. The single biggest win late in a long chat.
- Drop a size class. Qwen 3.5 0.8B instead of 2B lifts throughput from roughly 26 to 40 tokens per second on an iPhone 15 Pro.
- Let the phone cool. Sustained heavy generation throttles; a few minutes idle restores it.
- Turn off Low Power Mode.
- Turn off thinking mode for simple questions. Reasoning tokens are real tokens and they take real time — see Thinking Mode Explained.
- Shorten the prompt. This helps time-to-first-token specifically, not throughput.
And one recommendation that runs against instinct: if your current model already outruns your reading speed, try the next size up rather than down. You will likely lose nothing you can perceive and gain answers that are meaningfully better. Best Local LLM Models for iPhone compares the options.
Download Cloaked on the App Store and switch between 11 models on your own hardware — the only benchmark that matters is the one running on your device.
Frequently asked questions
How many tokens per second is fast enough?
Roughly 15 tokens per second — about 11 words — matches a comfortable reading pace, so text appears as fast as you can read it. Above 25 the difference is barely perceptible. Below 8 the pauses become distracting.
Why is my model slower than the benchmark said?
Benchmarks are usually measured on a cool device with a short prompt and nothing else running. Real use involves a warm phone, a long conversation with a large key-value cache, and background apps competing for memory bandwidth. A 30–40% gap between benchmark and daily use is normal.
Is time-to-first-token more important than tokens per second?
For short answers, usually yes. A model that starts in 0.3 seconds at 20 tokens per second feels faster than one that starts in 2 seconds at 40, even though the second finishes a long answer sooner. Perceived responsiveness is dominated by the initial wait.
Does a local model beat ChatGPT on speed?
On raw throughput, no — data centre GPUs are far faster. On time-to-first-token the gap narrows considerably, because a local model has no network round trip, no queue, and no rate limiting. On a poor connection, local wins outright.