Latency and Throughput Overhead of TEE-Protected Inference
SGX's memory ceiling makes confidential VMs the safer choice for large models.
A TEE builds a hardware-protected region where memory stays encrypted and code gets checked at every step. The host OS can't see inside it, and neither can the hypervisor or any other tenant sharing the same physical machine. Two families of CPU TEE matter for LLM inference, and once you put real load on them, they behave differently.
Intel SGX isolates at the process level: fine-grained, but boxed in by a fixed encrypted page cache, the EPC, sized around 92 to 128 MB. Push a working set past that ceiling and pages get evicted to regular memory, encrypted, then re-checked on every re-entry. For a 70B parameter model, that ceiling is a hard limit. Confidential VMs, Intel TDX and AMD SEV, take the opposite approach: they protect the whole guest rather than a single process. No enclave-size cap, multi-terabyte memory handled without strain, and none of the constant enclave-transition tax that drags SGX down. Anyone choosing between the two for a model above a few billion parameters is really choosing between a hard ceiling and a looser one; SGX should be treated as the exception, not the default, once model size crosses what EPC can hold.
GPU confidential computing, starting with NVIDIA's H100, adds a third layer on top of both. An on-chip Confidential Computing Engine encrypts every write to HBM with AES-256-GCM before the data leaves the chip, and the key never leaves the GPU's own security processor. CPU-to-GPU transfers over PCIe get encrypted too, and that transfer path accounts for most of the measured overhead. Before any of it runs, a remote-attestation handshake has to prove the enclave is genuine; That attestation handshake is paid once at provisioning rather than on every call.
Here's the pattern worth carrying through the rest of this piece: overhead piles up at memory boundaries and transition points (EPC paging, host-to-device transfers, enclave call crossings) and stays small inside the compute kernel itself. That's why overhead numbers swing so wildly across studies that all claim to measure "the same thing." Most of the confusion in this space comes down to someone quoting a percentage without saying which boundary it was measured across, and that single omission is responsible for more bad deployment decisions than any actual hardware limit.
What CPU TEEs cost at inference time for 7B–70B models
A September 2025 ETH Zurich study (arXiv 2509.18886) ran full Llama 2 inference inside Intel TDX and SGX. CPU TEE throughput overhead reached up to 10.68% for TDX across the configurations tested. Gramine-SGX landed at 4.80 to 6.15% overhead. TDX ran 5.51 to 10.68%, with the added cost over a plain VM measuring 3.02 to 7.01%. Turning on transparent huge pages kept TDX in the 4 to 10% band.
Here's what the ETH Zurich results show: SGX remained measurable even at large model sizes, which its tiny EPC would otherwise put in doubt. But look closer at the setup before drawing comfort from that. The study used a favourable EPC configuration; strip out that setup and SGX's EPC ceiling forces weights to spill into paged encrypted memory, degrading substantially. The broader research literature backs the same split: VM-based TEEs beat process-based TEEs like SGX specifically on I/O- and memory-heavy work, while SGX only stays viable for CPU-compute-bound tasks whose working set actually fits inside EPC. The real lesson is narrower than "SGX scales fine" suggests: it scales fine under one specific pairing that most production setups won't replicate by default.
Two levers keep CPU TEE overhead in single digits, per the ETH Zurich work: correct TEE configuration (huge pages, EPC tuning) and optimized matrix computation for the kernels inference leans on. For CPU-only confidential inference, that overhead is real but trades sensibly against the price of adding a GPU at all, and that tradeoff matters most for on-premises and edge deployments where a GPU isn't a given.
Where GPU TEE overhead actually lives on H100 (and why the published figures diverge so sharply)
NVIDIA's own figure for H100 confidential computing is 2 to 5% throughput overhead for most LLM inference workloads. That number gets cited constantly, and it isn't wrong, but the word "most" is doing more work than people give it credit for. Phala Network's benchmark work (arXiv 2606.23969) backs the low end: the GPU compute overhead is minimal, as the Phala Network B300 measurements confirm, with BF16 matmul throughput at 0.998 times non-confidential speed.
An IBM Research study presented at ICDCS 2025 tells a rougher story for model-swapping workloads on a single H100. Latency without confidential computing runs 20 to 30% lower than with it, throughput without confidential computing beats confidential mode by 45 to 70%, and GPU utilization runs roughly 50% higher outside CC mode. Set those numbers beside Phala's and the gap traces back to workload shape, not a flaw in either measurement. Anyone citing "H100 confidential computing costs 2 to 5%" without asking whether their workload swaps models is quoting the wrong study for their own deployment.
Here's the mechanism: a fixed, pre-loaded model moves its weight tensors across PCIe once, at startup, and every call after that is compute-bound, so the encryption cost amortizes down to almost nothing. A model-swapping workload re-encrypts large weight tensors from host DRAM into GPU HBM on every swap, so the workload stays data-transfer-heavy continuously instead of just at boot. That 45 to 70% throughput gap is that recurring cost, showing up call after call instead of once at the start.
A 2025 combined TDX-plus-H100 benchmark (arXiv 2607.19353) adds a wrinkle around concurrency. At concurrency 24, median time-to-first-token overhead runs 7.6% and end-to-end latency overhead runs 8.9%, both manageable. Push concurrency to 32, and TTFT overhead spikes to 725%, with end-to-end latency overhead hitting 86.2%, a jump the study ties to a scheduling regime change once the system crosses a saturation threshold. So the H100 overhead envelope comes down to three things: how often weights get transferred, how big the model is relative to HBM, and where concurrency sits relative to that scheduling cliff. A single cited percentage rarely represents a real deployment, and the workload's own shape has to get characterized first, before any number means anything.
What Blackwell's architecture reveals about where GPU confidential computing overhead is structurally located
A July 2026 Phala Network paper, "The Serialized Bridge" (Yin and Wang, arXiv 2606.23969), gives the clearest architecture-level look yet at where this overhead sits on NVIDIA's Blackwell generation, and the finding is almost too clean. GPU compute itself is barely touched: BF16 matmul throughput on the B300 measures at 0.998 times non-confidential speed, and GPU-local HBM traffic drops only marginally. The arithmetic was never the problem, and blaming the GPU's math units for confidential computing's cost just doesn't hold up.
The overhead sits structurally in what the paper calls the bridge, the encrypted staging path connecting the confidential VM to the GPU. Every host-to-device crossing pays a fixed setup cost of roughly 330 microseconds. Single-context host-to-device bandwidth falls to 0.203 times non-confidential speed, a severe hit, and device-to-host bandwidth falls to 0.211 times. That degradation traces almost entirely to the crossing itself, with only a minor contribution from the GPU's memory subsystem or compute units.
Software written for the non-confidential contract, meaning software that assumes transfers are cheap and instant, turns those bridge properties into large losses at the workload level. Dense-decode throughput drops 26%, KV-restore latency rises 131%, and model-load time balloons by a factor of 34: loading GPT-OSS-120B through the default safetensors path took 287.09 seconds, a number a CC-aware loader cut to 8.36 seconds by fanning weight shards across pooled CUDA contexts instead of moving them one at a time.
The vLLM async scheduling case shows how confidential computing can flip an optimization's sign entirely. Without CC, overlapping decode output transfer with next-step preparation saves about 3 milliseconds per step, a sensible win by any measure. With CC on, that same overlap costs about 4 milliseconds per step instead, because the overlapped copies now serialize against each other on the bridge instead of running side by side. The optimization never changed; the system around it did, and that alone flipped it against itself.
The Blackwell numbers generalize something already visible in the H100 data: overhead concentrates at the host-device interface instead of smearing across the GPU. Because it's concentrated there, most of it turns out to be recoverable through software that respects what the bridge actually does instead of fighting it.
How software changes recover most of the Blackwell overhead without touching hardware
That same Phala Network study makes the case that most of this overhead traces to software, not a hardware ceiling, and the recovery numbers back it up hard. A single scheduling flag change recovered 57% of the throughput gap, and a worker-thread patch pushed recovery to 92% in a qualified high-concurrency test. Neither change touched silicon at all, which is the point: the chip was never the bottleneck people assumed it was. Anyone blaming Blackwell's silicon for confidential computing overhead is pointing at the wrong layer of the stack.
The CC-aware model loader mentioned above works by restructuring how transfers cross the confidential bridge to avoid serialization. That's what took GPT-OSS-120B from a 287.09-second load down to 8.36 seconds. The same underlying principle — hiding encrypted data movement behind compute — appears in other lines of work in the confidential inference literature as well. Both approaches share the same underlying move: hide the encrypted data path behind compute instead of stacking it on top.
Across these recoveries, one pattern repeats: batch the crossings so the fixed 330-microsecond setup cost spreads across more data per call, pool CUDA contexts so transfers run in parallel instead of getting serialized through one default context, and turn off async scheduling optimizations that were built assuming I/O isn't serialized, because under CC, it is. Teams running inference on H100 or B300 in confidential mode should check their framework's default assumptions before taking a vendor's overhead figure at face value. Those defaults were almost never written with a serialized bridge in mind, and the gap between a default configuration and a tuned one can be the difference between a deployment that ships and one that gets killed on cost.
Partitioned TEE architectures that move most computation outside the enclave
Every layer of a large language model carries a different level of sensitivity, and partitioned TEE architectures take that seriously: they treat the model as a set of parts with different protection needs instead of one indivisible block. Instead of running the whole model inside the enclave, they run only the privacy-critical layers there and hand the bulk of the floating-point work to the GPU or CPU running in the clear.
Some partitioned-TEE approaches separate privacy-sensitive weights from the rest at training time, using a partition-before-training strategy: the backbone runs outside the enclave, the private segments run inside, with the goal of combining meaningful protection with manageable overhead. SecureInfer (arXiv 2510.19979, October 2025) applies a similar idea to LLaMA, routing each part of the model to the execution environment that matches its sensitivity and performance requirements, with the aim of keeping protected tensors secure without paying full TEE overhead on every computation.
Across the broader partitioned-TEE literature (SecureInfer, TwinShield, Amulet, ShadowNet among them), the pattern holds: 80 to 90%-plus of FLOPs pushed outside the enclave, speedups of 4 to 10 times over TEE-only baselines, and overhead as low as 1.2 to 2.8 times unprotected inference. AttestLLM, working with INT8 and INT4 quantization and full attestation included, reports overhead in the 16.8 to 19.6% range.
Partitioning buys real performance, but the tradeoff deserves stating plainly: it turns the security boundary into a design choice instead of an architectural guarantee. Someone has to decide which layers actually count as sensitive, and check that the partition doesn't leak information back out through activations computed outside the enclave. That's a genuine threat-modeling exercise, not a box to check off. And it's worth remembering why partitioning gets picked in the first place: for models at 7B parameters and up, fitting the full model inside a process-scoped TEE often just isn't possible given EPC limits. In plenty of deployments, partitioning is the only way the model fits in the box, not merely a performance trick.
How to read TEE overhead numbers against your actual workload before making a deployment decision
The measurements collected across this piece run from under 5% to several hundredfold, and that spread isn't noise or sloppy methodology. It's information about how sensitive different workloads are to where and how often data crosses a protected boundary. Pulling a single number out of that range without its context will mislead more than it informs. Treating a benchmark's headline percentage as if it transfers cleanly to a different workload shape is the mistake this piece has been arguing against since the first section, and it's worth saying plainly: the single most common error in evaluating TEE overhead is quoting NVIDIA's 2 to 5% figure for a workload that swaps models, when the number that actually applies there is closer to 45 to 70%.
Four variables decide where a specific deployment actually lands on that range. Model swap frequency matters enormously: a fixed, pre-loaded model amortizes transfer costs toward zero, while a multi-tenant setup that swaps models constantly keeps paying that 45 to 70% throughput penalty on every swap, per the IBM Research findings on H100. Model size relative to available EPC or HBM matters too, since it decides whether weights fit cleanly or start spilling into paged, re-encrypted memory. Concurrency matters, given how sharply overhead can jump once a system crosses a scheduling saturation point, the way TTFT overhead went from 7.6% to 725% between concurrency 24 and 32 in the combined TDX-H100 benchmark. And software maturity matters as much as the hardware generation itself: the Blackwell findings show a scheduling flag and a worker-thread patch recovering up to 92% of measured overhead, with zero change to the chip underneath.
Put those four variables together and a workload's real cost stops being a single percentage borrowed from someone else's benchmark. It becomes a curve, one that depends on how the deployment loads models, how many requests hit it at once, and whether the software stack was ever tuned for a confidential path to begin with. That's the actual engineering question anyone deploying this has to answer, and it gets answered by testing against the real workload, not by citing whichever published figure sounds best in a slide deck.
