- Home
- Benchmarks
On-device LLM benchmarks on iPhone
Real measurements from Pocket AI Lab's Build test, which downloads every catalog model, benchmarks it and deletes it again. 40 runs of 18 models on an iPhone Air, 2026-09-03 to 2026-10-23, app 1.1.0. The short version is on best local LLM for iPhone.
How fast do LLMs run on an iPhone Air?
The fastest model was LFM2 1.2B (GGUF) on llama.cpp at 77.3 tokens per second. 2B to 4B models without throttling landed between 26.3 and 36.6 tok/s, and 7B to 8B models dropped to 8 to 12 tok/s once the phone heated up.
| Model | Gen tok/s | pp512 / tg128 | Load s | Peak MB | Score | Date |
|---|---|---|---|---|---|---|
| LFM2 1.2B (GGUF) | 77.3 ± 0.4 | ~1,534 / 75.9 | 0.1 | 249 | 58.5 | 2026-10-23 |
| Qwen 3.5 0.8B (GGUF, Vision) | 73.7 ± 0.2 | ~2,241 / 72.8 | 0.3 | 598 | 46.6 | 2026-10-23 |
| Qwen 3.5 2B (GGUF, Vision) | 36.6 ± 1.3 | ~854 / 33.1 | 0.4 | 1,054 | 64.4 | 2026-10-23 |
| LFM2 2.6B (GGUF) | 35.9 ± 0.7 | ~560 / 33.1 | 0.1 | 266 | 60.2 | 2026-10-23 |
| Gemma 4 E2B (GGUF, Vision) | 35.0 ± 2.3 | ~708 / 32.8 | 2.6 | 1,490 | 63.6 | 2026-10-23 |
| Gemma 2 2B | 30.3 ± 0.3 | ~277 / 27.4 | 3.7 | 2,148 | 40.7 | 2026-10-22 |
| Llama 3.2 3B (GGUF) | 26.9 ± 0.6 | ~459 / 21.9 | 0.2 | 721 | 59.3 | 2026-10-22 |
| Qwen 2.5 3B | 26.3 ± 2.8 | ~249 / 22.4 | 1.7 | 2,360 | 55.1 | 2026-10-22 |
| Qwen2.5 1.5B (Core ML) | 3.0 ± 0.3 | ~646 / 2.9 | 39.4 | 4,354 | 24.6 | 2026-10-23 |
| Llama 3.2 3B | 23.9 ± 3.3 | ~183 / 19.9 | 2.1 | 2,568 | 59.3 | 2026-10-22 |
| Phi-3.5 Mini (GGUF) | 23.2 ± 1.2 | ~279 / 14.5 | 0.2 | 1,739 | 66.1 | 2026-10-23 |
| Gemma 3 4B (Vision) | 21.8 ± 4.2 | ~106 / 8.1 | 5.0 | 4,577 | 57.6 | 2026-10-22 |
| Gemma 3 4B (GGUF, Vision) | 20.7 ± 1.6 | ~314 / 10.9 | 2.2 | 2,175 | 60.2 | 2026-10-23 |
| Phi-3.5 Mini | 20.7 ± 2.8 | ~123 / 11.8 | 1.0 | 3,189 | 53.4 | 2026-10-22 |
| Qwen 3.5 4B (GGUF, Vision) | 14.7 ± 2.7 | ~158 / 5.1 | 0.4 | 1,252 | 68.6 | 2026-10-23 |
| Qwen2-VL 7B (Vision) | 11.6 ± 4.8 | ~46 / 4.8 | 2.5 | 5,461 | 63.6 | 2026-10-22 |
| Mistral 7B v0.3 | 9.1 ± 4.2 | ~44 / 4.4 | 1.1 | 4,671 | 51.7 | 2026-10-22 |
| Llama 3.1 8B | 7.6 ± 3.8 | ~42 / 4.5 | 2.7 | 5,121 | 57.6 | 2026-10-22 |
One row per model: the newest run without throttling, otherwise the newest run. Device iPhone Air (A19, 11.5 GB RAM as reported by iOS, iOS 26.6.1). Rows marked throttled slowed down while being measured, so their speed describes the hot phone more than the model.
What do the columns mean?
Gen tok/s is decode speed, the number you feel while reading an answer. The rest describes prompt reading, startup and memory.
- Gen tok/s is the mean of three 200-token generations after a warm-up, with the sample standard deviation.
- pp512 / tg128 follows the llama-bench convention: a fixed 512-token prompt and a 128-token generation. Prompt tokens are estimated, hence the ~.
- Load s is a cold load: whatever was resident is unloaded first, and this is the only measurement that isn't repeated.
- Peak MB is the highest
phys_footprintof the app from before the load until the run ends. - Score is the share of quality tasks passed, out of 100.
Is GGUF or MLX faster on iPhone?
Speed was close: Llama 3.2 3B ran at 26.9 tok/s on llama.cpp without throttling, and its MLX build measured 23.9 and 24.3 and 30.0 tok/s in three runs that both throttled.
The clear difference is memory. The same model peaked at 721 MB on llama.cpp and 2,568 MB on MLX. llama.cpp maps the GGUF file into memory, and iOS doesn't count those clean file-backed pages against the app the way it counts MLX's copied weights. One phone and one sweep isn't a verdict on either runtime, so run both on your own device.
How do the models score on quality?
Scores ranged from 24.6 to 68.6 on a 118-task suite. Instruction following separated the models most, and math was hard for all of them.
| Model | Math | Instructions | Knowledge | Retrieval | Multilingual | Score |
|---|---|---|---|---|---|---|
| Qwen 3.5 4B (GGUF, Vision) | 1/8 | 36/45 | 38/57 | 4/4 | 2/4 | 68.6 |
| Phi-3.5 Mini (GGUF) | 5/8 | 28/45 | 39/57 | 4/4 | 2/4 | 66.1 |
| Qwen 3.5 2B (GGUF, Vision) | 2/8 | 32/45 | 35/57 | 4/4 | 3/4 | 64.4 |
| Gemma 4 E2B (GGUF, Vision) | 3/8 | 32/45 | 33/57 | 4/4 | 3/4 | 63.6 |
| Qwen2-VL 7B (Vision) | 5/8 | 26/45 | 36/57 | 4/4 | 4/4 | 63.6 |
| LFM2 2.6B (GGUF) | 2/8 | 36/45 | 27/57 | 4/4 | 2/4 | 60.2 |
| Gemma 3 4B (GGUF, Vision) | 3/8 | 32/45 | 29/57 | 4/4 | 3/4 | 60.2 |
| Llama 3.2 3B (GGUF) | 1/8 | 35/45 | 27/57 | 4/4 | 3/4 | 59.3 |
| Llama 3.2 3B | 2/8 | 36/45 | 26/57 | 4/4 | 2/4 | 59.3 |
| LFM2 1.2B (GGUF) | 3/8 | 36/45 | 24/57 | 4/4 | 2/4 | 58.5 |
| Gemma 3 4B (Vision) | 1/8 | 30/45 | 30/57 | 4/4 | 3/4 | 57.6 |
| Llama 3.1 8B | 4/8 | 33/45 | 24/57 | 4/4 | 3/4 | 57.6 |
| Qwen 2.5 3B | 2/8 | 32/45 | 24/57 | 4/4 | 3/4 | 55.1 |
| Phi-3.5 Mini | 4/8 | 23/45 | 29/57 | 4/4 | 3/4 | 53.4 |
| Mistral 7B v0.3 | 2/8 | 27/45 | 25/57 | 4/4 | 3/4 | 51.7 |
| Qwen 3.5 0.8B (GGUF, Vision) | 2/8 | 26/45 | 20/57 | 4/4 | 3/4 | 46.6 |
| Gemma 2 2B | 3/8 | 19/45 | 19/57 | 4/4 | 3/4 | 40.7 |
| Qwen2.5 1.5B (Core ML) | 2/8 | 9/45 | 17/57 | 1/4 | 0/4 | 24.6 |
Suite v3: 118 tasks in five categories. Math covers arithmetic and word problems, Instructions checks format constraints, Knowledge asks factual questions, Retrieval answers from a long passage, Multilingual answers in another language.
How were these numbers measured?
Each model goes through the same fixed protocol inside the app: cool-down gate, cold load, warm-up, repeated speed probes, then the quality suite with greedy decoding.
- Gate. If the phone is already at the serious thermal state, the run waits up to 60 seconds to cool down.
- Load. Anything resident is unloaded, then the model loads from cold and the time is recorded.
- Warm-up. One short unmeasured generation, so the first measured run doesn't pay for Metal kernel compilation.
- Speed. A ~300-token prefill probe twice and a 200-token decode probe three times, then one pp512 / tg128 probe.
- Quality. Every task in a fresh conversation with its own timeout, temperature 0, top-k 1 and a fixed seed. Answers are graded by deterministic checks, never by another model.
A run is flagged as throttled when its decode speed fell more than 20% between its best and worst repetition, or when tg128 landed more than 20% below the decode mean. The thermal state is recorded too, but any iPhone reaches serious within minutes of sustained load, so it isn't used as the verdict.
The Build test also pauses to let the phone cool between models, and unloads each model so the next one starts from a clean footprint.
Can I run the same benchmark on my iPhone?
Yes. Model Benchmark in the app runs this protocol on any installed model and produces a report card you can share as an image or export as JSON.
Which runs are included?
10 Build test reports from app 1.1.0, all on the same iPhone Air.
- build-test-2026-09-04: 11 models, 2026-09-04
- build-test-2026-10-23: 18 models, 2026-10-23
- coreml-qwen2.5-1.5b-2026-09-06: 1 model, 2026-09-06
- llamacpp-gemma4-e2b-2026-10-22: 1 model, 2026-10-21
- llamacpp-lfm2-12b-2026-10-22: 1 model, 2026-10-21
- llamacpp-lfm2-26b-2026-10-22: 1 model, 2026-10-21
- llamacpp-qwen35-08b-2026-10-22: 1 model, 2026-10-21
- llamacpp-qwen35-2b-2026-10-22: 1 model, 2026-10-21
- llamacpp-qwen35-4b-2026-10-22: 1 model, 2026-10-21
- release-build-verification-2026-09-04: 4 models, 2026-09-04
All 40 runs
| Model | Gen tok/s | pp512 / tg128 | Load s | Peak MB | Score | Date |
|---|---|---|---|---|---|---|
| Gemma 2 2B | 29.7 ± 1.4 | ~272 / 24.7 | 3.7 | 2,130 | 56.5 | 2026-09-03 |
| Phi-3.5 Mini | 19.9 ± 2.6 | ~134 / 14.2 | 0.7 | 3,192 | 71.7 | 2026-09-03 |
| Llama 3.2 3B | 24.3 ± 3.7 | ~189 / 19.5 | 2.1 | 2,544 | 78.3 | 2026-09-03 |
| Qwen 2.5 3B | 24.5 ± 1.7 | ~241 / 21.7 | 1.2 | 2,345 | 69.6 | 2026-09-03 |
| Llama 3.1 8B | 8.2 ± 3.8 | ~43 / 5.0 | 3.1 | 5,137 | 80.4 | 2026-09-03 |
| Mistral 7B v0.3 | 8.6 ± 3.6 | ~45 / 5.1 | 1.9 | 4,641 | 65.2 | 2026-09-03 |
| Gemma 3 4B (Vision) | 22.2 ± 2.9 | ~122 / 12.8 | 5.1 | 4,559 | 76.1 | 2026-09-03 |
| Qwen2-VL 7B (Vision) | 11.5 ± 3.9 | ~46 / 5.3 | 3.5 | 5,464 | 78.3 | 2026-09-03 |
| Llama 3.2 3B (GGUF) | 27.2 ± 0.9 | ~459 / 24.7 | 8.9 | 696 | 78.3 | 2026-09-03 |
| Phi-3.5 Mini (GGUF) | 23.5 ± 1.2 | ~291 / 20.0 | 0.3 | 1,758 | 76.1 | 2026-09-03 |
| Gemma 3 4B (GGUF, Vision) | 21.5 ± 0.9 | ~391 / 17.2 | 2.3 | 2,149 | 82.6 | 2026-09-04 |
| Gemma 2 2B | 37.9 ± 0.5 | ~269 / 35.8 | 2.0 | 2,089 | 56.5 | 2026-09-04 |
| Phi-3.5 Mini | 23.3 ± 2.7 | ~94 / 14.7 | 1.0 | 3,167 | 71.7 | 2026-09-04 |
| Llama 3.2 3B | 30.0 ± 1.7 | ~181 / 17.8 | 1.2 | 2,508 | 78.3 | 2026-09-04 |
| Qwen 2.5 3B | 30.8 ± 2.3 | ~239 / 29.4 | 0.7 | 2,332 | 69.6 | 2026-09-04 |
| Qwen2.5 1.5B (Core ML) | 3.4 ± 0.2 | ~703 / 3.4 | 40.9 | 4,130 | 26.1 | 2026-09-06 |
| Gemma 4 E2B (GGUF, Vision) | 34.9 ± 1.2 | ~740 / 32.2 | 2.9 | 1,420 | 63.6 | 2026-10-21 |
| Qwen 3.5 0.8B (GGUF, Vision) | 73.8 ± 0.2 | ~2,272 / 73.1 | 0.3 | 550 | 46.6 | 2026-10-21 |
| Qwen 3.5 2B (GGUF, Vision) | 36.8 ± 1.3 | ~804 / 34.0 | 0.4 | 1,002 | 64.4 | 2026-10-21 |
| Qwen 3.5 4B (GGUF, Vision) | 15.1 ± 1.4 | ~236 / 6.4 | 0.6 | 1,207 | 68.6 | 2026-10-21 |
| LFM2 1.2B (GGUF) | 77.8 ± 0.3 | ~1,624 / 76.4 | 0.5 | 203 | 58.5 | 2026-10-21 |
| LFM2 2.6B (GGUF) | 34.9 ± 1.5 | ~528 / 32.9 | 0.1 | 220 | 60.2 | 2026-10-21 |
| Gemma 2 2B | 30.3 ± 0.3 | ~277 / 27.4 | 3.7 | 2,148 | 40.7 | 2026-10-22 |
| Phi-3.5 Mini | 20.7 ± 2.8 | ~123 / 11.8 | 1.0 | 3,189 | 53.4 | 2026-10-22 |
| Llama 3.2 3B | 23.9 ± 3.3 | ~183 / 19.9 | 2.1 | 2,568 | 59.3 | 2026-10-22 |
| Qwen 2.5 3B | 26.3 ± 2.8 | ~249 / 22.4 | 1.7 | 2,360 | 55.1 | 2026-10-22 |
| Llama 3.1 8B | 7.6 ± 3.8 | ~42 / 4.5 | 2.7 | 5,121 | 57.6 | 2026-10-22 |
| Mistral 7B v0.3 | 9.1 ± 4.2 | ~44 / 4.4 | 1.1 | 4,671 | 51.7 | 2026-10-22 |
| Gemma 3 4B (Vision) | 21.8 ± 4.2 | ~106 / 8.1 | 5.0 | 4,577 | 57.6 | 2026-10-22 |
| Qwen2-VL 7B (Vision) | 11.6 ± 4.8 | ~46 / 4.8 | 2.5 | 5,461 | 63.6 | 2026-10-22 |
| Llama 3.2 3B (GGUF) | 26.9 ± 0.6 | ~459 / 21.9 | 0.2 | 721 | 59.3 | 2026-10-22 |
| Phi-3.5 Mini (GGUF) | 23.2 ± 1.2 | ~279 / 14.5 | 0.2 | 1,739 | 66.1 | 2026-10-23 |
| Gemma 3 4B (GGUF, Vision) | 20.7 ± 1.6 | ~314 / 10.9 | 2.2 | 2,175 | 60.2 | 2026-10-23 |
| Gemma 4 E2B (GGUF, Vision) | 35.0 ± 2.3 | ~708 / 32.8 | 2.6 | 1,490 | 63.6 | 2026-10-23 |
| Qwen 3.5 0.8B (GGUF, Vision) | 73.7 ± 0.2 | ~2,241 / 72.8 | 0.3 | 598 | 46.6 | 2026-10-23 |
| Qwen 3.5 2B (GGUF, Vision) | 36.6 ± 1.3 | ~854 / 33.1 | 0.4 | 1,054 | 64.4 | 2026-10-23 |
| Qwen 3.5 4B (GGUF, Vision) | 14.7 ± 2.7 | ~158 / 5.1 | 0.4 | 1,252 | 68.6 | 2026-10-23 |
| LFM2 1.2B (GGUF) | 77.3 ± 0.4 | ~1,534 / 75.9 | 0.1 | 249 | 58.5 | 2026-10-23 |
| LFM2 2.6B (GGUF) | 35.9 ± 0.7 | ~560 / 33.1 | 0.1 | 266 | 60.2 | 2026-10-23 |
| Qwen2.5 1.5B (Core ML) | 3.0 ± 0.3 | ~646 / 2.9 | 39.4 | 4,354 | 24.6 | 2026-10-23 |