What on-device LLMs actually cost on mid-range Android

Almost every “local LLM on a phone” benchmark you see runs on an M-series Mac or a flagship Pixel. That’s not where most phones are. So I built a reproducible benchmark into NativeLM — my on-device document-chat app — and ran it on the hardware people actually carry: a mid-range Snapdragon 7 Gen 1 phone and a Snapdragon 870 tablet, both 8 GB, on Google’s LiteRT-LM.

One caveat up front, because it matters: these are CPU-backend numbers. GPU/NPU is the next step, so treat them as a floor, not a ceiling. They’re also from devices on a charger, so thermals aren’t ideal (more on that below). Every number here is exported straight from the in-app harness and committed, unedited, to the repo.

The numbers

Decode = steady-state tokens/sec (the speed you feel as the answer “types out”). Median of 3 runs, temperature 0.

Device	Qwen3 0.6B	Gemma 4 E2B	DeepSeek-R1 1.5B	Long-ctx TTFT	Peak RAM
Galaxy M55s — SD 7 Gen 1	3.3 tok/s	7.0 tok/s	—	~11 s	≤2.0 GB
Xiaomi Pad 6 — SD 870	4.9 tok/s	—	7.0 tok/s	~10 s	≤2.2 GB

For the document-chat (RAG) path, with the USE-Lite embedder: embedding runs at ~28 texts/sec, and an end-to-end retrieve() — embed the query, search the vector index, format the result — takes ~35–40 ms, near-identical on both chips.

Four things that surprised me

Smaller wasn’t faster. Qwen3 0.6B — the smallest model — was the slowest decoder on both devices, beaten by the 4× larger Gemma and DeepSeek. The community Qwen3 0.6B build looks unoptimised. On-device, the runtime build and quantisation matter far more than the parameter count on the box.

Decode is flat; prefill is not. Decode barely moves between a 10-word prompt and a 600-word one. But time-to-first-token jumps from ~1–3 s to ~11 s on the long document, because prefill reads the entire input before the first token appears. For grounded document Q&A, prefill — not decode — is the number users actually feel.

Retrieval is basically free; the model is the cost. Embedding a query and searching the index is ~35–40 ms. The LLM that then reads the retrieved passage is roughly 1000× slower. If on-device RAG feels slow, it’s the model’s prefill of the context, not the vector search. Optimise the model path, not the index.

Memory was the easy part. Peak process memory stayed ≤2.2 GB even for the 2.6 GB Gemma bundle — comfortable on an 8 GB device beside the OS. The constraint people fear (RAM) mattered less than the one nobody mentions (prefill latency).

The part I’m proudest of: not measuring it wrong

My first run reported ~3 tok/s for every model. The cause was a classic instrumentation trap: I was sampling process memory with Debug.getMemoryInfo() — which walks /proc/self/smaps, 100 ms+ on a multi-gigabyte process — on every token, inside the same coroutine collecting the token stream. The measurement serialised with token delivery and throttled the very thing it was timing. Moving memory sampling off the hot path was the fix.

Two more traps followed. The engine ignored my token cap and ran to end-of-sequence, so a reasoning model asked for “one word” happily emitted a 2,000-token <think> monologue and the run never finished — fixed by bounding generation in the harness (decode rate is independent of length, so a partial reply is plenty). And that deliberate stop surfaced as an error that looked like a crash, until I taught the harness that a cancel it initiated is a clean stop.

None of these are exotic. They’re the difference between a number you can publish and one that’s quietly 8× wrong.

Honest caveats

CPU backend only. The devices were plugged in to charge, and decode drifts with thermal load — Gemma measured 5.8 tok/s on a warm device and 7.0 once cooled, so I report the cooled figure and flag the spread. Energy-per-1000-tokens and a sustained 10-minute thermal soak aren’t measured yet; they’re next, alongside the GPU/NPU backends. And these are mid-range devices — genuinely low-end hardware is a separate target.

Reproduce it

It’s an in-app screen, not a lab rig: Settings → Developer → Benchmark, pick your downloaded models, tap Run, export JSON or CSV. The harness, the canonical prompt set, and the raw per-device exports all live in the repo — AGPL-3.0, no telemetry.

• The full method + tables: BENCHMARKS.md
• The numbers as a page: /benchmarks
• The harness + benchmark issue: litertlm-kmp#38

I build on-device and self-hosted AI — the engine is litertlm-kmp, the app is NativeLM — and I’m open to full-time and contract work. Happy to talk shop.