Encrypting Prompts and Responses in Transit to a TEE

TLS protects a prompt on its way to a server, but it does nothing once the prompt arrives. At that point the model has to read the text to do anything useful with it, so the data gets decrypted and sits in memory as plaintext, waiting to be acted on. This piece covers the mechanism that closes that gap: encrypting prompts end-to-end into a Trusted Execution Environment, so the data stays protected both crossing the wire and during the computation itself.

Here's the part most security writeups skip past: everything sitting in server memory during inference, the prompt itself, the model's intermediate reasoning state, any proprietary context stitched into the request, is readable by anyone with sufficient privilege on that machine. A cloud provider's system administrator could, technically, read it. Most providers forbid this in their terms of service, but policy is a promise, not a physical constraint, and the distance between "we promise not to look" and "we are unable to look" is exactly what this article is about.

That distance matters more for AI workloads than for most other computing tasks, because AI introduces failure modes that encryption-in-transit was never built to catch. Prompt leakage means someone reads the input at the point where it's decrypted for processing. Model extraction means an attacker infers proprietary weights just by watching how the model behaves under many queries, no server breach required, just patient observation. Training data inversion means reconstructing sensitive training examples from what the model outputs. These three failure modes don't involve intercepting a network packet; they happen at the layer where data sits in the clear so the processor can work with it, a state security researchers call protection "in use," distinct from the protection at rest and in transit that most compliance frameworks already assume is solved. This gap is structural, a limitation baked into how processors have always worked, rather than a bug waiting on a patch. Tightening firewall rules or rotating TLS certificates addresses the network layer, but the exposure lives in main memory during active computation, a different layer that takes different hardware to close, and that's the layer worth understanding in detail.

What a Trusted Execution Environment actually is and how it creates a protected computation boundary

A TEE is a region of the CPU and memory that's isolated from the rest of the machine, including the operating system running everything else on that same box. The isolation is physical: the processor itself generates and manages the keys that encrypt whatever memory gets assigned to the enclave, and that encryption happens below the level where an OS or hypervisor could interfere with it. A software convention that a clever administrator could route around would offer weaker guarantees than this.

Three properties fall out of that design, and all three matter specifically for AI privacy. Confidentiality means the host OS, the hypervisor, and the cloud operator running the physical hardware cannot read what's inside the enclave, even with root access to everything else on the machine. Integrity means nobody can quietly tamper with the code or data running inside while it executes. Remote attestation is the one that actually makes this useful for AI, letting a client, before sending anything sensitive, get cryptographic proof that a particular, unmodified piece of code is running inside a genuine TEE.

That third property is what separates hardware isolation from ordinary software sandboxing, and it's worth sitting with why. A software sandbox is enforced by an OS or hypervisor, both of which could themselves be compromised or coerced; a TEE's boundary is enforced by the processor itself, a different category of guarantee entirely. Code still runs in the clear inside the enclave, which is the whole point, since the model has to actually read the prompt to answer it. But anyone looking at that memory from outside the enclave, including the cloud provider that owns the physical server, sees nothing but encrypted bytes, which is exactly the property attestation lets a client verify before it commits anything sensitive. Attestation is what lets a user's device decide whether to trust that boundary before it sends a single token of a prompt.

The major hardware implementations and what they cover

Intel SGX was, for a long stretch, the reference implementation for high-security, data-intensive workloads, working at the level of individual process enclaves with fine-grained memory isolation. Intel TDX takes a different approach, extending protection to an entire virtual machine rather than a single enclave, and it's the technology behind Azure's confidential inferencing service as well as Apple's deployment on Google Cloud.

AMD SEV-SNP takes the whole-VM approach too, encrypting the entire memory footprint of a virtual machine, and it's generally available on Azure's confidential computing offerings. ARM TrustZone sits at the leaner end of the spectrum, built more for mobile devices and edge deployments than datacenter-scale inference.

The one that actually changes the math for large language models is NVIDIA's Confidential Compute, available on the Hopper and Blackwell GPU architectures, and this is worth paying the closest attention to. CPU-only TEEs were, for years, the real bottleneck: LLM inference runs on GPUs, and a CPU enclave that couldn't extend its guarantees to the GPU doing the actual matrix multiplication was solving only half the problem. It didn't matter how airtight the CPU enclave was if the tensor cores next to it were running in the clear. NVIDIA's H100 Confidential Compute reached general availability on Google Cloud in August 2025, and that timing matters, since it's roughly when GPU-based confidential inference stopped being a lab demo and started being something production teams could actually ship.

None of these implementations speak the same attestation language, and that's the detail glossed over whenever these chips get compared on a spec sheet as if they were interchangeable. Intel's quote format isn't AMD's, and neither maps directly onto NVIDIA's GPU attestation reports. Any system built to work across hardware vendors has to write attestation logic that knows exactly which chip it's talking to. That's a real engineering cost, not a footnote.

How end-to-end encrypted inference from client to TEE actually works

The core idea is simple to state even though the implementation has several moving parts: encryption terminates inside the enclave, not at the cloud provider's network edge. The provider's infrastructure never holds a key that could decrypt the prompt.

Everything starts with attestation, before a single byte of the actual prompt moves. The client sends the TEE a nonce, a random value that guarantees the response is fresh and not a replay of some earlier session, and the TEE responds with an attestation report describing its own state and platform, along with an ephemeral public key it generated for this session. A quoting enclave on the same physical machine signs that report into what's called a quote, and the client checks that quote against the hardware vendor's attestation service, Intel's, for instance, if the server is running SGX or TDX. Only once that check passes does the client start trusting the public key the enclave handed over.

That attestation step then has to bind itself to the actual encrypted connection, because a plain TLS handshake proves the channel is encrypted but says nothing about what code sits at the other end. This is where Attested TLS, or aTLS, comes in: it interleaves the attestation exchange with the TLS key exchange itself, so the key that signs the handshake is provably tied to a genuine TEE. The IETF has been standardizing exactly this in a draft called draft-fossati-tls-attestation, at version -10 and still active as of July 2026. In practice, the client's ClientHello message lists which attestation formats it's willing to accept along with a nonce, and the server hands back both its TLS certificate and an attestation report in the same handshake. The result is a connection that's simultaneously encrypted and attested: the client knows not just that nobody can read the wire, but specifically what software is running on the machine at the other end.

From there the actual prompt gets encrypted directly to the public key of the attested TEE, not to any infrastructure key the cloud provider controls. That distinction is the whole architecture in miniature. The provider's network layer only ever sees ciphertext, even as that ciphertext moves through the provider's own switches and load balancers. Apple's Private Cloud Compute documentation states this property outright: the request cannot be accessed by anything outside the validated nodes it was encrypted for.

Decryption, and the actual work of inference, happens only inside the enclave. The decryption key never leaves it: the host OS never sees it, the hypervisor never sees it, and the cloud operator running the physical rack never sees it either. Model weights get loaded into the enclave, the prompt gets decrypted there and there alone, and the response gets encrypted again, typically with authenticated encryption like AES-GCM, before it leaves the enclave boundary. The client holds the matching key on its end, so the response coming back is just as opaque to any intermediary as the prompt was going in, and attestation gets re-verified on every request too, not just once at session start, which matters if a long-running server could theoretically be tampered with between requests.

Real deployments that have already implemented this architecture

Apple's Private Cloud Compute is the most fully documented version of this pattern in production. User devices only send prompts to PCC nodes that can cryptographically prove they're running one of the specific software builds Apple has published to a public transparency log; the payload key gets wrapped only to the public keys of nodes whose measurements match an entry in that log. In July 2026, Apple extended this system to run on Google Cloud, using NVIDIA Blackwell GPUs, Intel TDX, and Google's own Titan security chip, marking the first time PCC operated outside Apple's own data centers. The stated property of that deployment is what's called zero-operator-access: cryptographic proof that no operator, including Google, can reach the inference data at any point in the pipeline. Google released open-source Prompt Encryption SDKs alongside that collaboration, meaning other teams can build a comparable end-to-end encrypted inference pipeline without inventing the attestation plumbing from scratch. Confidant AI is one example of a private AI assistant built on this kind of architecture from the ground up.

Microsoft's Azure Confidential AI Inferencing runs the LLM endpoint inside a TDX virtual machine and publishes a verifiable attestation policy naming exactly which model version, runtime, and system prompt version is approved to run. That policy is auditable by the customer directly, going beyond a compliance document's assertions from Microsoft. It's the difference between trusting a vendor's word and checking its work.

OpenPcc, an open-source research project from 2026, matches the property set of Apple's PCC and comparable confidential-inference approaches from Google and Anthropic, but runs on commodity confidential hardware and ships as a fully reproducible image: attestation logic, key exchange, all of it built from public source. That reproducibility is the interesting part. A research group, a regional cloud provider, or an enterprise security team doesn't have to take a single vendor's proprietary deployment on faith; they can build the equivalent themselves and check every piece of it.

Red Hat's work with Kata Containers points at where this is heading structurally. Sandboxed containers there automatically provision a confidential virtual machine and handle mutual attestation and decryption as a native step in the Kubernetes lifecycle, integrated rather than bolted on. That's a sign the pattern is becoming infrastructure, the kind of thing platform teams configure once, rather than a specialized feature only AI-focused vendors offer.

Performance overhead: what the benchmarks actually show

The overhead question used to have an ugly answer. Fully homomorphic encryption, the cryptographic alternative to hardware TEEs, generated roughly 0.2 tokens per second when applied to GPU inference, numbers nowhere close to production viable. That history is worth holding onto, because it's part of why TEE-based confidential inference looked, for years, like it might carry a similarly punishing tax.

The measured result tells a different story, and the gap between expectation and reality is the most underreported number in this entire space. A 2025 study out of ETH Zurich (Chrapek et al., arXiv:2509.18886) ran full Llama 2 inference pipelines, the 7B, 13B, and 70B parameter versions, inside both Intel TDX and SGX enclaves. Throughput overhead came in under 10%, latency overhead came in at around 20%, and turning on Intel's Advanced Matrix Extensions cut both figures further. One finding from that study cuts against intuition entirely: in certain configurations, CPU-based TEEs came out more cost-effective than GPU alternatives. That's backwards from what most people walk into this conversation expecting, since GPUs are supposed to be the fast option, and it's a good reminder that "confidential computing is slow" is closer to received wisdom than measured fact at this point.

GPU-based TEEs fare even better on paper. Inference on NVIDIA's H100 Confidential Compute showed throughput penalties in the low single digits, a gap that shrinks further as batch sizes and input lengths grow, which is exactly the direction production workloads tend to move as they scale.

So where does that leave the practical calculus? The overhead sits well within the range production systems already tolerate for other security controls, encryption at rest, network segmentation, audit logging, none of which are free either, which makes treating confidential inference as a mere research curiosity harder to justify today. What the benchmarks don't cover as thoroughly is multi-tenant orchestration at real scale, or what happens to overhead when aTLS handshakes stack up under heavy concurrent request volume. Those are open questions, not settled ones, and any team planning a production deployment should treat them that way rather than assuming lab numbers translate one to one.

What TEE-based encryption does not protect against

Side-channel attacks are the sharpest edge of this problem, and this is where anyone treating a TEE as an unbreakable vault needs to slow down. No amount of careful application code patches around a hardware-layer flaw. The tee.fail attack hits Intel SGX, Intel TDX, and AMD SEV-SNP all at once, because all three rely on a shared DDR5 timing design the attack exploits. Fixing it takes firmware-level patches, which underlines that the vulnerability lives in the hardware itself, not in how any particular vendor implemented its software stack on top of it. This isn't a new category of problem, either, since Meltdown and Spectre established years ago that hardware isolation boundaries can leak information through timing behavior even when nobody breaches the boundary directly. TEEs inherit that same risk class; they don't erase it.

Insecure code inside the enclave is a separate failure mode, and arguably the more common one in practice, if less dramatic. Academic analysis of a broad set of open-source TEE projects found that roughly a third of them bypass the official SDK's cryptographic APIs in favor of rolling their own, and a similar share show generally insecure coding habits. That matters because TEE hardware guarantees the confidentiality and integrity of whatever code is running; it makes no promise about whether that code was written correctly. A TEE protects bad code just as thoroughly as it protects good code. That's a colder way of putting it, but it's the accurate one, and it's the point most vendor marketing quietly skips over.

Attestation gaps show up specifically in VM-based TEEs. Because these operate at the level of a whole virtual machine rather than a narrow enclave, they can end up implicitly trusting the cloud provider in ways that don't line up with an attestation flow meant to exclude the provider entirely. Getting that binding right, tying the attestation report unambiguously to the state actually running, without quietly trusting the provider along the way, is genuinely hard, and how well it's done depends heavily on the specific deployment.

None of this touches what happens at the model layer itself, and this is a mistake worth naming directly: a TEE offers no defense against a model being manipulated. The OWASP LLM Top 10 for 2025 lists system prompt leakage (LLM07:2025) as its own distinct risk, and for good reason. A TEE closes the channel where plaintext might leak in transit or in memory, but a cleverly worded adversarial input can still talk a model into revealing its own system prompt, and no amount of channel encryption touches that. If that system prompt contains business logic, secrets, or the model's own safety instructions, an attacker doesn't need to break any encryption at all; they just need the right question. Channel security and model security are different problems, solved by different controls, and treating TEE deployment as a substitute for prompt-injection defenses is a category error. It's the single most common misconception in how these systems get pitched.

Then there's the plain operational weight of running any of this. Attestation flows don't map cleanly onto a standard Kubernetes deployment; teams need custom tooling, or a framework like Kata Containers that's built the attestation lifecycle in from the start. That complexity is a deployment risk in its own right, separate from whether the underlying cryptography is sound. A system can be mathematically airtight and still fail in production because nobody wired the attestation checks into the deploy pipeline correctly.

Complementary approaches and where TEEs sit among them

Fully homomorphic encryption is, on paper, the cleanest answer to all of this: computation happens directly on ciphertext, and the data is never decrypted at any point, not even during inference. That's a genuinely stronger guarantee than a TEE offers, since a TEE still decrypts data inside the enclave, even if nothing outside can see it happen. The trouble is that FHE, at current performance levels, produces only a tiny fraction of a token per second when applied to GPU-scale LLM inference. Call it a long-term research direction, not something a team ships today for generative AI workloads.

Multi-party computation splits a computation across several parties so no single one of them ever holds the complete plaintext. Some recent work combines MPC with differential privacy, an approach sometimes called SPRINT in the research literature, but it comes with a real constraint: the model has to be fine-tuned specifically before it can be served this way, which rules out dropping it into an existing inference pipeline without extra engineering work. MPC tends to shine on narrow, well-defined tasks rather than as a general substitute for serving an arbitrary LLM.

Set against both of those, TEEs occupy a middle position that's easy to underrate precisely because it isn't theoretically perfect, and correcting that undervaluation is the whole point of this section. FHE promises more on paper; MPC fits certain narrow tasks well. TEEs are the only one of the three actually running production traffic today, at overhead numbers, per the ETH Zurich benchmarks, that a real system can absorb without redesigning its whole serving stack around the cryptography. The tradeoff is that a TEE asks an operator to trust a piece of hardware and its vendor's implementation, and that trust carries exposure to side-channel research still actively evolving. That's a real cost, but it's a smaller one than betting a production system on FHE throughput that hasn't arrived yet. What's actually running at acceptable overhead right now carries more weight than theoretical purity, and that judgment call is the one this piece keeps landing on.

One consideration sits underneath all three approaches and gets less attention than it deserves: none of the encryption or attestation signatures involved here are immune to a future where quantum computing becomes practical. The schemes protecting TEE memory today, and the signatures underlying attestation quotes, will eventually need replacements built to withstand quantum attacks. Some key management vendors have already started adding post-quantum algorithm support ahead of that transition, which is the right instinct. Any organization building serious infrastructure on top of TEE-based inference right now should treat post-quantum migration as a line item on the roadmap, not a distant hypothetical to deal with later.