Point your phone at a handwritten note, a menu in a language you don’t read, a screenshot of an error message, or a photo of the contents of your fridge, and ask a question about it. A vision-capable model answers.
The interesting part is where that happens. With most assistants, the image travels to a server. With an on-device multimodal model, it doesn’t leave the phone — and that changes what kinds of photos it is reasonable to ask about.
This article sits under How LLMs Actually Run on Your iPhone.
How a language model ends up “seeing”
A language model works on vectors — lists of numbers representing meaning. Text becomes vectors through tokenization and embedding. The trick behind multimodal models is that images can become vectors in the same shared space.
The pipeline has three parts:
1. The vision encoder. The image is resized and divided into a grid of patches — commonly 14×14 or 16×16 pixels each. A vision transformer processes these patches, producing a vector per patch that encodes what is there and how it relates to everything else in the frame. A 448×448 image becomes on the order of a thousand vectors.
2. The projection layer. Those vision vectors live in the encoder’s own space, which isn’t the language model’s space. A learned projection maps them across, so an image patch and a word end up as compatible representations.
3. The language model. It now receives a sequence containing both image-derived vectors and your text tokens, and processes them together with the same attention mechanism. From its perspective, the image is simply more context.
That is why you can ask “what’s unusual about this photo?” and get a sentence back. There is no separate image module producing labels that get handed to a text module — the same network reasons over both.
The Qwen 3.5 family implements this at sizes small enough to run on a phone, which is a genuinely recent development.
What images cost you in context
An image is not free. Those thousand-odd patch vectors occupy the model’s context window exactly like text tokens do.
A single high-resolution image can consume 1,000–2,000 tokens of context — the equivalent of a fairly long article. Consequences worth knowing:
- Several images in one conversation fill the window quickly.
- After sending a few photos, earlier messages may drop out.
- Higher resolution costs more context, which is why apps downscale before encoding.
- Time-to-first-token is longer for image prompts, because encoding happens during prefill.
Practically: attach the image you’re actually asking about, ask what you need in that conversation, then start a new one for the next image.
What small vision models do well
Tested against realistic use, the strong cases are:
Description. What is in the frame, roughly where, and what is happening. Reliable.
Text extraction. Reading signs, menus, receipts, labels, handwritten notes, screenshots. Genuinely useful, and combined with a multilingual model it makes a decent offline translator for anything printed — see Offline AI for Travel.
Screenshots. Error messages, unfamiliar interfaces, charts. Screenshots are clean, high-contrast, and text-heavy, which suits these models well.
Object identification. Plants, tools, ingredients, components — at a general level.
Visual questions. “Is this ripe?”, “which of these connectors is USB-C?”, “what does this symbol mean?”
And where they fall down
Being straight about this matters more than the marketing.
Dense small text. A page of fine print will produce errors. Verify anything that matters.
Precise counting. Counting many similar objects in a cluttered scene is unreliable across model sizes, and worse when small.
Fine spatial detail. Exact positions, small differences between similar images, precise measurements.
Specialist domains. A general vision model has no business interpreting an X-ray, a rash, or a legal document’s significance. It will produce a confident, fluent, potentially wrong answer. Do not use it this way.
Faces. Recognizing specific individuals is not what these models do, and it is not a gap that should be closed.
The pattern is consistent: strong on “what is this and what does it say,” weak on “exactly how many, exactly where, exactly which.”
The metadata problem nobody mentions
Here is the part that gets skipped in almost every discussion of AI and photos.
A photo from a phone is not just pixels. It carries EXIF metadata, which typically includes:
- GPS coordinates — often accurate to a few metres
- Exact date and time
- Device make, model, and sometimes a unique identifier
- Camera settings and orientation
So a photo of a whiteboard is also a record of the building you were in and when. A photo of a document is a record of where that document was.
When an image is uploaded to a cloud service for analysis, all of that goes with it unless the service strips it — and whether it does, and what it retains, is a policy question you have to trust rather than verify. AI Apps That Don’t Collect Your Data covers how to check claims like this.
On-device processing changes the shape of the problem. Cloaked strips EXIF and GPS data locally before the image reaches the model, so location metadata is neither analysed nor retained in the conversation. The image itself never leaves the device, because there is nowhere for it to go — the privacy architecture has no upload path.
That distinction is what makes it reasonable to photograph a bank statement, a prescription label, a contract, or a page of a diary and ask a question about it. Not because a promise was made about deletion, but because transmission never occurs.
Practical use
Downscale first when you can. Fewer context tokens, faster encoding, and for most questions no loss of useful information.
Ask specific questions. “What does the third line say?” beats “tell me about this image.”
Verify anything consequential. Extracted numbers, dates, and amounts should be checked against the source.
Prefer screenshots to photographs of screens. Cleaner input, better results.
Use a larger model for harder images. Qwen 3.5 2B handles everyday cases well; 4B and 9B are meaningfully better on dense or ambiguous images if your device can hold them.
In Cloaked, all four Qwen 3.5 models are vision-capable, from 622MB up to 5.6GB — so on-device image understanding is available even on modest hardware. Best Local LLM Models for iPhone compares them.
Download Cloaked on the App Store — photograph anything you like and ask about it, with no upload, no retention, and no metadata leaving your phone.
Frequently asked questions
How does an AI model see an image?
A vision encoder divides the image into patches and converts each into a vector — the same format the model uses for word tokens. The language model then processes image and text representations together, which is why it can answer questions about a picture in ordinary sentences.
Can a small on-device model really understand photos?
For description, text extraction, object identification, and general questions, yes. Small vision models are noticeably weaker at fine detail — dense small text, precise counting in cluttered scenes, and specialist domains like reading medical scans, which they should not be used for.
Do photos get uploaded when I ask an AI about them?
With cloud assistants, yes — the image is transmitted to a server to be processed. With on-device vision, no. The image is read into memory, encoded, and analysed by a model running on your phone. Nothing is transmitted, which is why it works in airplane mode.
What about GPS data in my photos?
Photos usually carry EXIF metadata including precise GPS coordinates, timestamp, and device details. Cloaked strips EXIF and GPS data on-device before the image is processed, so location data is not retained even locally in the conversation.