Benchmarks · litertlm-kmp
On-device LLM benchmarks
What private, on-device LLMs actually cost on real mid-range Android — not a MacBook or a flagship Pixel. Measured from inside the app on Google's LiteRT-LM, CPU backend, fully offline. A floor, not a ceiling: GPU/NPU is next.
Device matrix
| Device | SoC | RAM | Android |
|---|---|---|---|
| Samsung Galaxy M55s SM-M558B · phone · mid-range | Snapdragon 7 Gen 1 | 7062 MB | Android 16 |
| Xiaomi Pad 6 23043RP34I · tablet | Snapdragon 870 | 7610 MB | Android 14 |
arm64-v8a · 8 cores · LiteRT-LM CPU backend (6 threads).
LLM performance
Decode is steady-state tokens/sec — the speed you feel. TTFT is time to first token (prefill latency). Median of 3 runs after a warm-up; temperature 0, fixed seed; generation bounded to 128 tokens for a stable rate.
Galaxy M55s — Snapdragon 7 Gen 1
| Model | Cold load | Peak RAM | Decode | TTFT (short) | TTFT (long-ctx) |
|---|---|---|---|---|---|
| Qwen3 0.6B | 807 ms | 1701 MB | 3.3 tok/s | 3082 ms | 11508 ms |
| Gemma 4 E2B | 1227 ms | 2005 MB | 7.0 tok/s | 1272 ms | 11001 ms |
Xiaomi Pad 6 — Snapdragon 870
| Model | Cold load | Peak RAM | Decode | TTFT (short) | TTFT (long-ctx) |
|---|---|---|---|---|---|
| Qwen3 0.6B | 772 ms | 1799 MB | 4.9 tok/s | 2418 ms | 9726 ms |
| DeepSeek-R1 Distill 1.5B | 1103 ms | 2170 MB | 7.0 tok/s | 819 ms | 10805 ms |
RAG performance
The document-chat path: how fast the embedder turns text into vectors, and how long an
end-to-end retrieve() takes (embed the query → vector search → format).
| Device | Embedder | Embed throughput | End-to-end retrieve() |
|---|---|---|---|
| Galaxy M55s | USE-Lite (100-dim) | 28.7 texts/s | 35 ms |
| Xiaomi Pad 6 | USE-Lite (100-dim) | 28.5 texts/s | 40 ms |
What the numbers say
- Smaller isn't faster. Qwen3 0.6B — the smallest model — is the slowest decoder on both devices. On-device, the runtime build and quantisation beat parameter count.
- Decode is flat; prefill is not. Decode barely changes with prompt length, but TTFT jumps to ~10–11 s on a ~600-word prompt. For long-document Q&A, prefill is the number users feel.
- Retrieval is cheap; the model is the cost. RAG retrieve() is ~35–40 ms — roughly 1000× cheaper than the LLM that reads the retrieved context.
- Memory is modest. Peak process memory stays ≤2.2 GB even for the 2.6 GB Gemma bundle — comfortable on an 8 GB device.
Method & honesty
The harness loads each model into the real engine and times its actual token stream; peak RAM is sampled off the timing path. Numbers here mirror the canonical BENCHMARKS.md and the raw per-device JSON exports in the repo.
These runs were on devices plugged in; decode drifts with thermal load (Gemma measured 5.8 tok/s warm vs 7.0 cooled — the cooled figure is reported). Prefill tok/s, energy per 1000 tokens, a sustained thermal soak, and the GPU/NPU backends are not measured yet — they're next, against this CPU baseline.