Running a large language model on a phone sounds like it shouldn’t work. These are the same architectures that need warehouses of GPUs — and yet a 5.6GB model generates fluent text on a device that fits in a pocket, with no connection to anything.
It works because of three things that arrived at roughly the same time: quantization made the weights small enough, Apple Silicon’s unified memory made them addressable, and frameworks like MLX made the computation efficient enough to be practical. This guide explains the whole path — from the moment you hit send to the moment a word appears.
If you want the shorter version first, On-Device AI: The Complete Guide covers the concepts at a higher level. This article is the mechanical explanation underneath it.
The one distinction that explains everything: training vs inference
Almost every “how can a phone possibly do this” question dissolves once you separate two very different jobs.
Training is how a model is created. It means running trillions of words through a network, measuring how wrong each prediction was, and nudging billions of parameters a tiny amount in the right direction — repeated for weeks or months across thousands of GPUs. This genuinely requires a data centre. Nobody trains models on phones.
Inference is using a model that already exists. The parameters are fixed. Your prompt gets converted to numbers, multiplied through those fixed parameters, and the result is a probability distribution over what word comes next. Pick one, append it, repeat.
Inference is enormously cheaper than training. It is still not free — a 2B-parameter model performs about 4 billion arithmetic operations per token generated — but it is the kind of arithmetic a modern phone GPU does comfortably. The A17 Pro and later chips handle trillions of operations per second.
The whole on-device AI category exists in the gap between those two numbers.
The three things that have to fit in memory
When people ask whether a model “fits” on a phone, they usually mean storage. Storage is the easy part. The real constraint is memory, and three separate things compete for it.
1. The weights
The model’s parameters — the numbers learned during training. This is the bulk of it. A model described as “2 billion parameters” has 2 billion of these numbers, and how much space they take depends entirely on how precisely each one is stored.
At full 16-bit precision, 2 billion parameters is about 4GB. Quantized to roughly 4 bits each, the same model is about 1.2–1.6GB. This is why quantization is not an optimization detail — it is the thing that makes the category possible at all. LLM Quantization Explained covers how the compression works and what it costs in quality.
2. The KV cache
This is the part most explanations skip, and it is the reason long conversations behave differently from short ones.
A transformer generates one token at a time, and each new token attends to every token before it. Recomputing all of that history for every single word would be brutally slow, so the intermediate results — the keys and values — get cached.
The catch is that the cache grows linearly with conversation length. A short exchange might use 50MB. A conversation that fills a 32,000-token context window can use well over a gigabyte on a larger model. That memory is not optional and it is not reclaimable while the conversation is live. Context Windows Explained covers the practical consequences.
3. Working memory
Every layer produces intermediate activations that exist briefly during the forward pass. Modest compared to the other two, but not zero — and iOS is unforgiving. If an app’s memory footprint spikes past what the system will allow, the app is terminated, not slowed down.
The practical consequence: required memory is far larger than the download. Cloaked’s catalogue states a minimum for each model, and the ratio is striking — the 622MB Qwen 3.5 0.8B needs 5GB, the 2.9GB Qwen 3.5 4B needs 10GB, and the 5.6GB Qwen 3.5 9B needs 12GB. The minimum has to cover the peak of a long conversation with an image attached, not the idle case.
Unified memory: why this works on Apple Silicon specifically
On a traditional PC, the CPU has its own RAM and the graphics card has separate VRAM. To run a model on the GPU, the weights must be copied across a bus into VRAM. That copy is slow, and VRAM is usually much smaller than system RAM — which is why desktop enthusiasts obsess over whether a model “fits in VRAM.”
Apple Silicon does not work that way. The CPU, GPU, and Neural Engine all address the same physical memory. There is one pool, and every processor can read from it directly.
For language model inference this is close to ideal, because inference is memory-bandwidth-bound rather than compute-bound. Generating a single token requires reading essentially every weight in the model. Not clever selective access — all of it, every time. Modern chips can do the arithmetic far faster than they can fetch the numbers, so the ceiling is how fast weights move from memory to the processor.
This explains an otherwise strange observation: a 1B model doesn’t feel twice as fast as a 2B model in a simple way — throughput tracks how many bytes must be read per token. Halve the bytes, roughly double the speed. It is also why quantization improves speed as well as footprint. Fewer bits per weight means fewer bytes to move.
Apple publishes the architectural detail in its Apple Silicon developer documentation.
Where the computation actually happens
An iPhone has three processors that can do this work, and MLX uses all of them.
The GPU does most of the heavy lifting. Transformer inference is dominated by large matrix multiplications, which is exactly what GPUs are built for. In Cloaked, the majority of generation time is GPU time.
The Neural Engine (ANE) is a fixed-function accelerator built for specific neural network patterns at very high efficiency. It is dramatically more power-efficient than the GPU for the operations it supports, but it is less flexible — not every operation in a modern transformer maps onto it cleanly.
The CPU handles tokenization, sampling, control flow, and anything that doesn’t vectorize well. Small share of the time, but on the critical path.
The scheduling between them is the framework’s job, and it is genuinely difficult work. What Is Apple MLX? goes into how Apple’s framework approaches it.
What happens between your tap and the first word
Here is the actual sequence, in order.
-
Tokenization. Your text is split into tokens — sub-word chunks. “Unbelievable” might become
un,belie,vable. Each maps to an integer ID. English averages roughly 0.75 words per token. -
Embedding. Each token ID becomes a vector of several hundred to several thousand numbers, positioning that token in a learned semantic space.
-
Prefill. The entire prompt runs through every layer of the network at once. This is the fastest phase per token, because all prompt tokens are processed in parallel — and it is what populates the KV cache. On a long prompt you may notice a brief pause here before anything appears.
-
Generation. Now it goes one token at a time. Each pass produces a probability distribution across the whole vocabulary; a sampler picks one token; that token is appended and fed back in. Repeat until the model emits a stop token or hits a limit.
-
Detokenization. Token IDs convert back to text, streamed to the screen as they arrive — which is why answers appear word by word rather than all at once.
Steps 4 and 5 loop dozens of times per second. Everything in that loop happens inside the device. There is no step where text is packaged and transmitted, because there is no destination — which is the architectural point behind Cloaked’s privacy architecture.
Why speed changes during a conversation
Two effects, and knowing them makes local model behaviour predictable rather than mysterious.
The KV cache grows. Every token added to the conversation means more cached history for each new token to attend to. Generation slows gradually as a conversation extends. It is rarely dramatic in normal use, but a conversation approaching the context limit will noticeably lag the same model at the start of a fresh chat.
Thermal throttling. Sustained inference generates heat. iOS responds by reducing clock speeds to protect the device. A model producing 40 tokens per second on a cool phone may settle to 25–30 after several minutes of continuous heavy generation. This recovers on its own; it is the system working correctly, not a defect.
Both effects are covered in more depth in Tokens Per Second Explained, including why published benchmark numbers rarely match what you see.
What actually determines quality
Given fixed hardware, three things move quality — and they are not equally important.
Parameter count matters most, but with sharp diminishing returns. The jump from 0.6B to 2B is transformative. From 4B to 9B is real but far subtler, and mostly visible on tasks with many reasoning steps.
Training data quality frequently beats raw size. Microsoft’s Phi models are the standing proof: trained heavily on curated synthetic textbooks and code rather than scraped web text, Phi-4 Mini competes with substantially larger models on technical work. Meanwhile a 2026-vintage 2B model outperforms 2023’s 7B models on most benchmarks — same size class, three years of better data and better recipes.
Inference-time reasoning is the newest lever. Models with a thinking mode spend extra tokens working through a problem before answering, which reliably improves multi-step accuracy at the cost of latency. Thinking Mode Explained covers when it earns its keep and when it just makes you wait.
Choosing a model with the mechanics in mind
Everything above collapses into a fairly simple decision.
Your device RAM sets the ceiling, and the app enforces it. 4GB devices run the lightweight tier, 6GB adds the mid-size multimodal models, 8GB adds Phi-4 Mini, and the 5.6GB flagship needs 12GB — which in practice means an iPad or a recent Pro iPhone. Which iPhones Can Run Local AI has the full matrix.
Your task sets the floor. Quick questions, summarizing, rewriting, and translation are well served by 1–2B models. Multi-step reasoning, longer code, and nuanced analysis benefit from 4B and up.
Your patience is the third axis, and it is underrated. A 2B model answering in eight seconds often beats a 9B model answering in thirty — the smaller model gets used, and the larger one quietly stops being opened.
This is why Cloaked ships 11 models from 5 labs rather than one. Qwen 3.5 0.8B at 622MB is the recommended default: vision-capable, thinking-capable, and light enough that heavy turns stay clear of the memory ceiling. From there you scale up or down against your own hardware and habits. Best Local LLM Models for iPhone works through the specific trade-offs model by model.
The part that isn’t a trade-off
Everything above involves compromise. Smaller models are less capable. Bigger models are slower. Longer conversations cost memory. These are real constraints and pretending otherwise would be dishonest.
The privacy property is the exception, because it isn’t a setting — it’s a consequence of the architecture. When inference happens in unified memory on your device, there is no request to log, no retention window to configure, and no policy to trust. A comparison of on-device and cloud AI makes the practical difference concrete.
The engineering is genuinely impressive. The privacy is just what’s left over when you remove the server.
Download Cloaked on the App Store to run any of these models on your own device — free, no account, and fully functional in airplane mode once a model is downloaded.
Frequently asked questions
How can an iPhone run an AI model when it needs a data centre?
Training needs a data centre; running a trained model does not. Training a model involves thousands of GPUs processing trillions of words over months. Running one — inference — means multiplying your prompt through weights that already exist. That is a much smaller job, and quantization shrinks the weights enough to fit in phone memory.
Does running an LLM locally drain the battery?
It uses meaningful power while generating, and almost none when idle. In Cloaked, a 2B model producing a 500-word answer typically costs 1–3% of an iPhone battery. Generation is bursty rather than continuous, so a normal conversation costs far less than an hour of video playback.
How much RAM do I need to run a local LLM?
Far more than the download size, because the KV cache, vision encoder, and working memory all have to fit alongside the weights. In Cloaked, the 622MB Qwen 3.5 0.8B requires 5GB, the 1.6GB Qwen 3.5 2B requires 6GB, the 2.9GB Qwen 3.5 4B requires 10GB, and the 5.6GB Qwen 3.5 9B requires 12GB.
Is on-device inference slower than ChatGPT?
Usually yes on raw tokens per second, but the gap is smaller than people expect because there is no network round trip. A local 2B model starts producing words in well under a second, while a cloud model spends 200–800ms on the request before it generates anything.
Does the Neural Engine do all the work?
No. Apple MLX distributes work across the GPU, Neural Engine, and CPU depending on the operation. The GPU handles most of the large matrix multiplications in transformer inference; the Neural Engine excels at specific fixed patterns. The unified memory architecture is what makes splitting the work practical.