On-Device AI: The Complete Guide to Running LLMs on Your iPhone

Running AI on your iPhone means a language model is installed locally and all inference — the actual thinking — happens on your device’s chip, never touching a server. Apple’s MLX framework makes this practical on modern iPhones. You get real AI capabilities with zero data leaving your phone, no internet required, and no account needed.

This guide covers how on-device inference actually works, which iPhones can handle it, how to pick the right model, and what the honest trade-offs are compared to cloud AI. By the end, you’ll know exactly what to expect — and whether it’s right for you.

Here’s what we’ll cover:

• How on-device inference works and what makes it possible now
• The iPhone hardware requirements and what different chips can handle
• Apple’s MLX framework and why it changed everything
• How to choose the right model for your device and use case
• The real trade-offs between on-device and cloud AI
• Privacy: what “on-device” actually guarantees

How On-Device Inference Works

On-device inference is the process of running a neural network’s forward pass — the computation that produces output — entirely on local hardware. No data is sent to a server. The model weights live on your device’s storage, get loaded into memory, and the chip does all the math.

For large language models (LLMs), inference is a token-by-token process. The model reads your input and predicts the next word, then the next, then the next — until it decides to stop. Each step requires multiplying thousands of matrices. That’s why LLMs were assumed to require data center GPUs until recently.

What changed is hardware. Modern smartphone chips — especially Apple’s A-series — have Neural Engine accelerators that can perform trillions of operations per second. The iPhone 15 Pro’s A17 Pro Neural Engine runs at 35 TOPS (tera-operations per second). That’s enough throughput to run a 3B-parameter model at 15–25 tokens per second, which is faster than most people read.

The other key breakthrough is model quantization. Full-precision neural networks store each weight as a 32-bit float. Quantized models compress those weights to 4 bits or 8 bits with surprisingly little quality loss. A 3B parameter model that would need 12GB at full precision drops to under 2GB at 4-bit quantization — small enough to fit on a phone with room to spare.

iPhone Hardware: What Your Device Can Handle

Not every iPhone can run LLMs well. The bottleneck is unified memory — Apple Silicon shares memory between the CPU, GPU, and Neural Engine, which matters because LLM weights need to stay in memory during inference.

Here’s a practical breakdown by chip generation:

A15 Bionic (iPhone 13 series, iPhone 14, iPhone 14 Plus)

The A15 has 6GB of total system memory on most configurations. That leaves roughly 4–5GB available for apps after the OS takes its share. You can run 1B–2B parameter models comfortably. The 3B models work but may cause memory pressure that slows other apps. The Neural Engine delivers 15.8 TOPS.

A16 Bionic (iPhone 14 Pro, iPhone 14 Pro Max)

A modest step up in Neural Engine performance. The same 6GB memory constraint applies, but thermal management improved, so sustained inference runs longer before throttling. Practical ceiling is similar to A15: 1B–3B models.

A17 Pro (iPhone 15 Pro, iPhone 15 Pro Max)

This is the first chip Apple built on a 3nm process. The Neural Engine jumps to 35 TOPS — more than double the A15. Apple also bumped RAM to 8GB on Pro models. This is the sweet spot for on-device AI. Qwen 3.5 4B, for example, produces output at roughly 16 tokens per second on an iPhone 15 Pro.

A18 and A18 Pro (iPhone 16 series)

The A18 further closes the gap between on-device and cloud. The A18 Pro in iPhone 16 Pro models has 8GB of RAM and enough Neural Engine headroom to run 7B models at usable (if not fast) speeds. According to Apple’s benchmarks, the A18 Pro performs 15% faster than the A17 Pro on machine learning workloads.

The practical rule: If you have an iPhone 14 Pro or newer, you can run productive on-device AI. If you have an iPhone 15 Pro or newer, it’s genuinely good.

RAM, not storage or chip generation, is what decides whether a model will load. Which iPhones Can Run Local AI has the full device-by-device matrix and the enforced RAM requirement for every model.

Apple MLX: The Framework That Made This Possible

Before MLX, running LLMs on Apple Silicon meant using frameworks designed for other hardware — frameworks that copied data back and forth between CPU and GPU memory constantly. On a Mac or iPhone where CPU and GPU share the same memory pool, that’s wasted work.

Apple MLX is an open-source machine learning framework built from the ground up for Apple Silicon’s unified memory architecture. Instead of treating CPU, GPU, and Neural Engine as separate devices that pass data between them, MLX treats them as a single compute target sharing one memory space. The result is significantly less overhead and faster inference.

MLX also introduces lazy evaluation — computations are queued and fused where possible, reducing the number of passes through memory. For matrix multiplications (the core of LLM inference), this is a meaningful speedup.

The MLX project on GitHub is actively maintained by Apple’s machine learning research team and the community around it has grown quickly. The MLX Community on Hugging Face has converted hundreds of open-source models into MLX format, including quantized variants optimized for iPhone memory constraints.

For a deeper look at the framework itself, see our post on what Apple MLX is and why it matters.

One underappreciated advantage of MLX: because it’s Apple’s own framework, it gets first-class integration with future hardware improvements. Every time Apple releases a new chip, MLX can immediately take advantage of new capabilities without waiting for third-party framework updates.

For a step-by-step walk through the whole inference path — tokenization, prefill, the KV cache, and where the computation actually lands on the chip — see How LLMs Actually Run on Your iPhone.

Choosing the Right Model for Your iPhone

With 11 models available in different sizes and specializations, picking the right one isn’t obvious. The core tension is between capability and resource cost — bigger models understand more nuance and follow complex instructions better, but they need more memory and produce tokens more slowly.

The Size vs. Speed Trade-Off

A useful mental model: every doubling of parameter count roughly doubles memory use and halves inference speed, in exchange for a meaningful jump in quality. A 1B model is fast but limited. A 7B model is capable but slow on most iPhones. The 3B–4B range is where most users land.

Here’s a rough guide to what each size handles well:

Sub-2B models (Qwen 3 0.6B, Qwen 3.5 0.8B, Llama 3.2 1B, Qwen 3 1.7B, DeepSeek R1 1.5B): Fast and light, and all run on 4–5GB of RAM. Good for quick questions, simple rewrites, translation, and drafting short content. Less reliable on complex reasoning or long multi-step instructions.

2–3B models (Qwen 3.5 2B, Llama 3.2 3B, Ministral 3 3B): The practical sweet spot for most iPhones, needing 6GB of RAM. Handles everyday tasks well — writing assistance, explaining concepts, code help, Q&A — and Qwen 3.5 2B adds image input.

4B models (Qwen 3.5 4B): The best all-around model in the catalogue. Strong reasoning, good instruction following, multilingual, and vision-capable. Requires 10GB of RAM, so it is limited to recent Pro iPhones and M-series iPads.

Flagship (Qwen 3.5 9B): Best quality, but demanding — 5.6GB to download and 12GB of RAM to run. Around 10 tokens per second on an iPhone 15 Pro and roughly twice that on an M-series iPad Pro, where it belongs.

Specialized Models

Not all models are built for general chat. DeepSeek R1 1.5B reasons step by step before answering, which makes it better at math, logic puzzles, and coding problems, and the Qwen 3 and 3.5 families offer the same behaviour as a toggle. Qwen models have stronger multilingual capabilities than the Llama series. If you primarily work in a language other than English, Qwen is worth trying first.

For a detailed breakdown of which models perform best for specific use cases, see the best local LLM models for iPhone.

On-Device vs. Cloud AI: An Honest Comparison

It would be dishonest to claim on-device AI beats cloud AI at everything. It doesn’t. Here’s where each approach genuinely wins.

Where cloud AI wins:

• Raw capability — GPT-4 class models have hundreds of billions of parameters. Even the best 7B model isn’t in the same league for complex reasoning, creative work, or nuanced instruction-following.
• Context length — cloud models often support 128K+ token context windows. Most on-device models work best under 8K tokens.
• Speed at large scale — cloud providers run thousands of GPUs in parallel. For very long outputs, a cloud model might actually finish faster.

Where on-device AI wins:

• Privacy — your input never leaves your device. Period. Not a policy promise, a physical constraint.
• Availability — no internet needed after download. Works in airplane mode, underground, in areas with no signal.
• No account, no API key, no subscription — download and use.
• Latency for short responses — a local model starts producing output in under a second, with no network round-trip.
• Cost — no per-token charges. Run as many requests as you want.
• Consistency — no server outages, no rate limits, no model version changes without your consent.

The right framing isn’t “which is better” but “which is right for this conversation.” Many conversations don’t need GPT-4 class capability. They need a knowledgeable, private assistant that responds quickly and doesn’t store what you said. For those — which is most everyday tasks — on-device AI is not a compromise.

For a deeper look at this trade-off, see our full comparison of on-device AI vs. cloud AI.

What “On-Device” Actually Guarantees for Privacy

The privacy benefit of on-device AI isn’t just a feature — it’s an architectural guarantee. Understanding why matters, because plenty of apps claim to be “private” without the same guarantees.

When you use a cloud AI service, your input travels to a server. That server runs the model, generates a response, and sends it back. Even if the company has good privacy policies, the data existed on external hardware at some point. It was transmitted over a network. It could have been logged for debugging. It could be used for model training. These aren’t accusations — they’re descriptions of how the architecture works by default.

On-device inference removes those attack surfaces entirely. Your input never leaves the app’s process on your device. No network request is made for inference. The model doesn’t phone home. There’s nothing to intercept in transit, nothing stored on a server, no training pipeline that could incorporate your data.

This matters for conversations you wouldn’t want an audience for — medical questions, financial situations, relationship issues, legal concerns, career discussions. These are exactly the conversations people have with AI assistants, and exactly the ones that deserve the strongest privacy protection.

A few honest caveats: if an on-device AI app includes optional cloud-connected features — web search, sync, cloud backup — those specific features involve network requests. What stays local is the model inference itself. In Cloaked, for example, the optional DuckDuckGo web search makes a search query; your AI conversation and model output do not.

Frequently Asked Questions

Can modern iPhones actually run large language models?

Yes. iPhones with A16 Bionic chips or newer have enough unified memory and Neural Engine throughput to run models between 1B and 7B parameters at usable speeds. The iPhone 15 Pro and 16 series handle 3B–4B parameter models comfortably. Larger models like 7B run but are noticeably slower, typically 5–10 tokens per second.

What is Apple MLX and why does it matter for on-device AI?

MLX is Apple’s open-source machine learning framework built specifically for Apple Silicon — the same chip architecture used in recent iPhones and Macs. It treats CPU and Neural Engine memory as a single pool, eliminating the memory-copy bottleneck that slows other frameworks. This is what makes running LLMs on iPhone practical rather than just technically possible.

Do on-device AI apps work without an internet connection?

Yes, once a model is downloaded. The model file is stored on your device and all inference happens locally. You need internet only for the initial download. Apps like Cloaked include features like DuckDuckGo web search that are optional — if you use web search, that one query goes out; everything else stays local.

How much storage do on-device AI models take?

It depends on the model. Qwen 3 0.6B is 317MB — less than many photos. The recommended default, Qwen 3.5 0.8B, is 622MB. Mid-range models like Llama 3.2 3B are about 2GB, and the Qwen 3.5 9B flagship is 5.6GB. Required RAM matters more than storage: the 9B model needs 12GB of memory to run.

Is on-device AI as capable as ChatGPT or Claude?

Not for all tasks. Cloud models have far more parameters and compute behind them. But for everyday tasks — drafting text, answering questions, explaining concepts, coding help, summarizing — a well-quantized 4B model is genuinely useful, not just a demo. The trade-off is capability for privacy, and for many conversations that trade-off is entirely worth it.

Start Running AI Locally

On-device AI went from a research demo to something you can use in daily life in a remarkably short time. The hardware is there. The framework — MLX — is there. The models are there, open-source, and getting better with every release.

The remaining piece is an app that packages this into something that doesn’t require reading GitHub READMEs to get running.

Cloaked is an iOS app built for exactly this. It ships with 11 open-weight models from Alibaba, Microsoft, Mistral AI, Meta, and DeepSeek. You pick a model, download it once, and have a private AI assistant that works in airplane mode, requires no account, and processes everything on your device. Voice input, web search via DuckDuckGo, project-based organization, and persistent memory — all local.

If you’ve been curious about on-device AI but haven’t had a reason to dig in, this is the right time. The technology has crossed the threshold from interesting to genuinely useful.

Download Cloaked on the App Store