The on-device AI vs cloud AI choice is not a preference, it's an architecture decision with a bill attached. Here's the framework for picking, and the hybrid pattern most production apps end up on.
CodeKrypt Bot 5 min readSomewhere in the first architecture conversation for a mobile AI feature, someone asks the on-device AI vs cloud AI question, and the room treats it like a preference — like choosing a design system. It isn't. It's a decision that sets your latency budget, your inference bill, your offline behaviour, and how much custom mobile engineering the project needs, months before a single user sees the feature.
Small on-device models are having a real moment in 2026 — Apple's on-device frameworks and Google's AICore-based Gemini Nano have made "run it on the phone" a credible default rather than a niche optimisation. That makes the decision harder to skip past, not easier, because now there's a real choice to get wrong.
On-device AI means the model — usually a small, quantized model in the hundreds-of-millions to low-billions of parameters range — runs on the phone's own silicon, through a framework like Apple's Core ML or Android's AICore, or via an open-weights runtime such as MLC-LLM or llama.cpp compiled for mobile. Cloud AI means the app sends a request over the network to a hosted model — GPT, Claude, Gemini, or your own fine-tuned endpoint — and waits for a response.
The two options aren't different flavours of the same thing. They fail differently, cost differently, and demand different engineering:
| Dimension | On-device | Cloud API |
|---|---|---|
| Latency | Sub-100ms, no network round trip | Typically 300ms–2s+ depending on model and connection |
| Cost structure | High upfront engineering cost, near-zero marginal cost per inference | Low upfront cost, billed per request/token forever |
| Model capability | Small, quantized models — good at narrow, bounded tasks | Access to frontier models for complex reasoning and long context |
| Offline support | Works without a network connection | Fails or degrades without connectivity |
| Data privacy | User data never leaves the device | Request payload transits and is processed on a third party's servers |
| Iteration speed | Model updates ship through app releases or a separate download pipeline | Swap models or prompts server-side, no app update needed |
| Engineering burden | Quantization, per-OS integration, device-matrix testing | API integration, rate limits, cost monitoring |
Neither row is a tiebreaker on its own. The right call depends on what the feature actually needs to do and how often.
On-device makes sense when a feature is used often, needs to feel instant, or touches data that shouldn't leave the phone. Autocomplete, smart replies, on-device semantic search over content the user already has, classification and tagging, and single-document summarisation are all bounded enough for a small quantized model to handle well, and frequent enough that per-request cloud billing would add up fast. Health data, biometric processing, and anything with a strict data-residency requirement point the same direction — not sending the data anywhere is the simplest way to satisfy that constraint.
The catch is that "wins" here assumes steady, high-volume usage that justifies the upfront cost of shipping and maintaining the model. A feature ten people use once a week doesn't recoup that investment.
Cloud wins when the task needs reasoning quality a phone-sized model can't deliver — multi-step agents, long-context retrieval, anything where a wrong answer is expensive. It also wins when usage is low or unpredictable, because you're not paying for idle capacity, and when you need to iterate on the model or prompt weekly without shipping a new app build. If the feature is genuinely new and you don't yet know if it'll be used enough to justify on-device investment, start in the cloud and measure.
In practice, the choice usually isn't one or the other for the whole app — it's per feature, sometimes per request. A lightweight on-device model or classifier handles the common case and decides, in milliseconds, whether it can answer confidently or needs to hand off. Anything outside its bounds — an ambiguous query, a task requiring more context than the small model can hold — routes to a cloud model. The user gets instant responses for the frequent path and full model quality for the hard path, without paying cloud costs on every single interaction.
The two paths front-load cost differently, and that changes what a realistic MVP budget looks like. A cloud-only build gets a feature in front of users faster because there's no model to quantize, package, or test across a device matrix — but every request is a recurring bill that scales with adoption, so a successful feature can get expensive precisely because it worked. An on-device build costs more up front in mobile engineering — model conversion, per-OS integration (Core ML and AICore are not the same work twice), and a way to update the model outside the app store review cycle — but that cost doesn't grow with usage.
Decide this per feature, not once for the whole app: what does it need to do, how often, and what happens if the network isn't there. That's the conversation to have before committing engineering time, not after the first cloud bill arrives. It's the first thing we scope on AI mobile app projects, because it determines the mobile architecture more than any other single decision.
AI agent memory cost is the budget line most teams discover only after their context window balloons. What naive context injection actually costs, what tiered retrieval saves, and real 2026 pricing to plan against.
AI agent observability cost is the line item most teams forget until an incident forces it. What evals and monitoring actually cost, broken into tooling, compute and human review, with real 2026 figures.
Quality is the top-cited reason AI agent pilots stall, yet barely half of teams run evaluations at all. What an AI agent evaluation framework actually has to check — trajectory, not just output — and how to build one before you scale.