On-Device AI Inference on Apple Silicon for Privacy-Preserving Applications

Running a real AI model on a phone used to mean fighting the phone itself. Traditional chip layouts give the CPU, the GPU, and any dedicated accelerator their own separate pools of memory, and every time a model's weights need to hop from one pool to another, you pay for it twice: once in latency, once in battery. That copying overhead is basically why cloud inference won by default. Phones had plenty of compute sitting around, sure, but that compute was walled off from the memory it needed to touch, on every single layer, for every single forward pass.

Apple's unified memory architecture just, well, removes the wall. CPU, GPU, and Neural Engine all read from the same physical memory pool, so weights get loaded once and shared across whatever compute unit needs them next. Copying, duplicate allocation, thrashing between pools — none of it happens anymore. A large model on a phone stops being a contradiction in terms and becomes something closer to a normal engineering problem, one you can actually chip away at.

That also changes what "the bottleneck" even means. For token generation specifically, the limiting factor was never how fast the chip multiplies matrices; it's how fast it can pull weights off memory in the first place. Bandwidth decides how quickly the next word shows up, not raw compute. The M4 offers 120 GB/s of unified memory bandwidth, and the M5 pushes that to 153 GB/s, a 28% jump. Apple's own numbers put the resulting token generation speedup at somewhere between 19 and 27% in practice. Those figures aren't decoration; they're the actual reason a chat response on an M5 MacBook feels like a conversation instead of a buffering wheel.

Here's the part that's easy to skate past but is really the whole reason this section comes first: bandwidth improvements erase the performance excuse that's been sitting there for a decade. Developers who wanted to build something privacy-respecting still had to route inference to a server, because on-device execution was too slow to be usable for anything beyond a toy demo. Take that excuse away, and you have to ask what's left justifying the cloud round-trip. Not much remains, for a lot of workloads now.

What the Neural Engine actually does, and what the headline TOPS figures obscure

The Neural Engine sits apart from the CPU and GPU, a dedicated block built for the matrix multiplication that neural network math is mostly made of. Apple has shipped Neural Engines across its silicon lineup, and the climb since then is honestly kind of startling: the A13 Bionic in 2019 hit 5 TOPS, the A18 Pro in 2024 reaches 35 TOPS. Call it sevenfold in under five years, entirely on-device, no server in the loop at any point.

The M5 keeps that curve going with a 16-core Neural Engine, and Apple claims over 4x the peak GPU compute of the M4, with new GPU Neural Accelerators handling the big matrix multiplications that dominate LLM inference specifically. Impressive on its face, but a careful reader should slow down here and ask what these TOPS numbers are actually counting, because it's not quite what it sounds like.

Apple markets the M4's Neural Engine at 38 TOPS. That number follows an industry habit of counting INT8 operations at double the rate of FP16, on the theory that lower-precision math runs proportionally faster. On Apple's Neural Engine, though, INT8 inputs get converted back to standard precision before the computation actually happens, so that doubling never shows up in reality. Measured peak throughput lands around 19 TFLOPS, about half the headline figure.

Is that dishonest marketing? I don't think it's quite that simple. INT8 quantization still saves real bandwidth, because smaller weights move through memory faster, and given that bandwidth is the actual constraint (see above), that savings is genuine even if it's not the savings the TOPS number implies. Call it a bandwidth benefit wearing a compute benefit's clothes. For a developer sizing up inference latency, the distinction isn't academic: assume the 2x compute multiplier is real, budget your quantization scheme around it, and your latency estimates will be off by a wide margin.

Even at the corrected 19 TFLOPS, that's dedicated neural compute running on a chip drawing laptop-level power, which is still a genuinely good result. The caveat here doesn't undercut the case for on-device inference at all; it just tells you where the gains are actually coming from, which is the distinction an engineer needs and a press release tends to quietly skip.

How Apple compresses a capable language model into ~3B parameters that fit and run well on-device

Bandwidth and compute don't matter if the model itself doesn't fit in the box. Apple's on-device foundation model runs at roughly 3 billion parameters, and it was sized and trained for the memory and compute profile of Apple Silicon from the start, not shrunk down from something bigger as an afterthought once someone noticed it didn't fit.

A capable general-purpose model at full precision blows through a phone's memory budget without much effort. So the real engineering question is: how much quality survives when you compress that hard? Apple leans on two techniques working together. First is 2-bit quantization-aware training, where the model's weights get compressed down to 2-bit precision and the training process accounts for that loss from the very beginning, rather than training at full precision and quantizing afterward as a lossy afterthought. The model learns to be good at 2 bits, rather than merely surviving the conversion down to it.

Second is KV-cache sharing: a two-block architecture where the second block reuses key-value caches the first block already computed. That alone cuts memory usage by 37.5%, and on a device with a fixed, fairly modest memory budget, that's the gap between a feature that ships and one the OS kills for eating too much RAM.

Apple layers on low-bit palettization too, mixing 2-bit and 4-bit LoRA adapters so the system fine-tunes for specific tasks without stashing a full-precision copy of the weights for every single capability. That's how one device ends up supporting several distinct AI features without each one demanding its own multi-gigabyte model file sitting on disk.

The upshot: Apple's compressed model holds up against open models of similar parameter count running at full precision, which means compression here is preserving reasoning quality rather than just shrinking a file and hoping nobody notices the difference. On an M5 MacBook Pro, time-to-first-token comes in under 10 seconds for a dense 14B architecture and under 3 seconds for a 30B mixture-of-experts model. Stated plainly, those numbers might sound almost boring, but they're the line between "on-device AI" as a marketing phrase and on-device AI as something a person actually sits down and uses without getting impatient. There's a privacy consequence tucked in there too: a model that's genuinely good enough removes the temptation, for developer and user both, to punt a sensitive query up to a bigger cloud model the moment the local one stumbles.

The developer frameworks that make on-device inference the default path, not the difficult one

All this hardware and model work is wasted if building on it is miserable. So the fair question is: how much friction does Apple actually put between a developer and shipping something on-device?

Not much, it turns out. The Foundation Models framework, introduced at WWDC 2025, gives developers a native Swift API that reaches the on-device model in as few as three lines of code. Inference through it costs nothing, no per-token billing, which quietly removes the single biggest financial reason teams default to cloud APIs in the first place. An education app can generate quizzes from a student's own notes; an outdoors app can run natural language search with zero signal. Neither one needs a network call to work.

Core AI, announced for WWDC 2026 as the successor to Core ML, is built specifically for LLMs and generative AI running entirely on-device. It routes work across CPU, GPU, and Neural Engine through one unified API, so a developer isn't manually deciding which chip handles which layer of the model. It offers a memory-safe Swift interface with zero-copy data paths and fine control over inference memory, and it shifts model compilation to happen ahead of time, off the user's device entirely, so what the user experiences at runtime is a near-instant load rather than a compile step disguised as a loading spinner.

Apple now frames the developer stack in three layers, and each earns its spot. Core ML handles classical, non-neural machine learning: decision trees, tabular models, the unglamorous workhorse stuff that's still everywhere. Core AI takes the neural networks and transformers. MLX and MLX Swift sit out at the research edge, for developers bringing their own weights, training something from scratch, or fine-tuning what Apple didn't ship.

The performance numbers out of MLX are worth a developer's attention on their own merits. MLX runs 20 to 87% faster than llama.cpp for generation on Apple Silicon, for models under 14B parameters. Ollama, when it switches to the MLX backend, sees 93% faster decode with zero configuration changes required, which is the kind of free lunch that almost never shows up in this business. On quantization, Q4_K_M has become something of a sweet spot: about 3.3% quality loss for a 75% reduction in model size. That's the tradeoff that turns a model that wouldn't fit into one that fits comfortably with room to spare.

One more thing worth flagging, because it's genuinely interesting: Orion, published in March 2026, is the first open system for programming the Neural Engine directly, bypassing Core ML's private APIs entirely. It hit over 170 tokens per second running GPT-2 124M inference on an M4 Max, and along the way documented 20 separate hardware constraints on the Neural Engine, 14 of which nobody outside Apple had written down before. Sit with that for a second, because there's headroom on this chip that the public frameworks haven't even touched yet. The ceiling for on-device inference on this hardware is probably higher than anything currently shipping actually shows.

Put together, you get a mature toolchain with a clear three-layer structure, documented tradeoffs, and a ceiling that keeps climbing. For a growing share of workloads, on-device isn't the harder path anymore; it's just the easier one.

Where Private Cloud Compute fits when on-device genuinely isn't enough (and what privacy properties it preserves)

None of this should get oversold, though. A 3-billion-parameter model, however cleverly compressed, cannot handle every task a reasonable person might throw at it. Pretending otherwise would be bad engineering wearing a marketing costume, and to Apple's credit, they don't pretend otherwise.

Private Cloud Compute is the answer for whatever falls outside the on-device model's reach. It runs cloud inference on custom Apple Silicon under a hardened operating system, built around the explicit goal that data sent to PCC stays inaccessible to everyone, Apple's own engineers included. Five requirements define what that actually means in practice: computation is stateless, so requests get processed ephemerally in memory with nothing written to persistent storage; privacy guarantees are enforced by the system itself rather than a policy document someone could quietly ignore; no privileged runtime access exists, so even Apple's own operators can't reach into user data mid-process; the system is non-targetable, meaning it can't be pointed at a specific user's request to single them out; and the whole security model is meant to be verifiable from the outside rather than just asserted from the inside.

That last one is where Apple went further than most companies bother to. It opened a Virtual Research Environment so outside researchers could actually inspect the system rather than take Apple's word for any of it. That's not a small gesture; it's an open invitation to people with every incentive to find a flaw and every ability to publish it.

The five core requirements are designed to remain constant even as the underlying infrastructure evolves. That tells you something about how Apple thinks about this system: the privacy properties are the invariant, and the physical infrastructure is just an implementation detail underneath. Worth noting, too, that leaning on a second company's infrastructure brings a new set of trust assumptions along with it, and that's a thread worth pulling on as the arrangement matures rather than something to just wave through.

Taken as a whole, the on-device model handles what it can, and PCC picks up the overflow under the same structural commitments. At neither tier does user data land in the kind of conventional logging-and-retention system that turns a breach or a subpoena into a privacy disaster.

What this architecture means for developers and users who need AI without surveillance (and where Confidant fits)

Zoom out and it's the same pattern at every layer, just applied differently each time. Unified memory kills the bandwidth penalty. The Neural Engine, read honestly instead of through its headline TOPS number, delivers real dedicated compute. The 3B model compresses hard without gutting quality. The frameworks turn building on all of it into a low-friction decision instead of a specialist's side project. None of these are separate features bolted onto a keynote slide; they're one stack, built so capable AI inference doesn't require handing a user's data to a server in the first place.

For developers, that stack knocks out both the economic argument and the complexity argument for defaulting to the cloud. Free inference through Foundation Models, one unified API through Core AI, a research-grade path through MLX for anyone who needs it: on-device is, for a lot of real workloads now, just the path of least resistance.

For users, the part that actually matters isn't a toggle buried three menus deep in Settings, or a sentence in a privacy policy nobody reads. It's structural: an inference path that can't log or transmit data it was never handed in the first place. Architecture does the work here, not anyone's goodwill.

Here's the honest caveat, one that shouldn't get smoothed over just because the rest of the piece is complimentary: this entire architecture assumes Apple hardware, and that's a real floor, not a footnote. Plenty of users, and pretty much every enterprise with a mixed fleet, don't live in an all-Apple environment, and that caps how far this particular guarantee reaches on its own.

That's roughly the gap Confidant sits in. It starts from the same conviction running through everything described above: privacy has to be decided into the architecture before the system gets built, not bolted on afterward to satisfy a compliance checklist someone drew up in a different meeting. For someone who wants AI that's actually capable without handing their data to a server they'll never see or audit, Confidant tries to carry that same principle through, without requiring the user to become their own ML infrastructure engineer just to get there.

The claim this whole piece has been building toward, section by section, is a fairly simple one once you've walked through all of it: there's no law of physics forcing a choice between AI that's genuinely capable and AI that's genuinely private. That tradeoff is a decision engineers make, one way or the other. Apple Silicon's stack, from unified memory up through Private Cloud Compute, is what it looks like when that decision gets made deliberately, in one direction, and actually followed through to the end.