Encrypting Prompts and Responses in Transit to a TEE
Hardware-sealed enclaves protect LLM prompts from cloud operators and hypervisors during inference.
Encrypting a prompt in transit to a trusted execution environment is a layered protocol: client-side encryption, an attested key exchange, and hardware-enforced decryption that happens only inside a sealed enclave, and each of those steps closes a specific gap that standard transport security leaves wide open. The sections below walk through why that gap exists and how the protocol closes it, piece by piece.
Why standard TLS leaves cloud LLM prompts exposed
A cloud LLM service protected only by standard TLS leaves the prompt fully readable in memory the moment inference starts, because TLS was built to secure a wire, not a computation. Blaxel's explainer on trusted execution environments puts the problem in plain terms: during inference, the model weights, the customer's prompt, and the agent's reasoning state all sit in memory as plaintext, and any privileged user with access to that memory can read all of it. TLS does its job up to the server boundary and then stops. Past that point, the cloud operator, the hypervisor underneath the virtual machine, and any co-tenant sharing the same physical host can potentially observe the data in use.
The usual answer to this, software isolation through virtual machines or containers, does not actually close the gap. A compromised hypervisor has full access to guest memory, and a single kernel vulnerability can expose every container running on that host. Nor is this a hypothetical concern confined to academic worst-case scenarios. A 2026 paper from Beihang University and Renmin University of China examined LLM API routers, the components that terminate one TLS session from the user and immediately open a fresh one upstream to the model, and found that these routers hold the entire interaction in plaintext as a structural feature of how they work, making them an application-layer man-in-the-middle by design rather than by accident. The organizations carrying the most exposure from this are the ones with the least room for error: healthcare, legal, and financial enterprises that route sensitive documents through cloud LLM services every day. For those users, the prompt itself sits exposed at the one moment it matters most: while the model is actually reading it.
What a TEE guarantees
A trusted execution environment is a hardware-isolated region of a processor whose confidentiality guarantee comes from silicon, not from a software configuration that an administrator could quietly change or an attacker could quietly break. The processor itself enforces the isolation boundary, so no software running outside the TEE, not the operating system, not a hypervisor, not an administrator with root access, can inspect or modify what runs inside it. The cryptographic keys that make this possible are provisioned into tamper-resistant hardware at manufacturing time and never leave the CPU's secure boundary. Blaxel's explainer makes the practical implication concrete: even someone with root access to the host machine cannot see the model weights or the customer's prompt once they're loaded into encrypted TEE memory.
What makes this guarantee distinct from ordinary encryption is the layer it protects. Encryption at rest protects a file sitting on disk. TLS in transit protects data crossing a network. Neither protects data while a program is actively computing on it, and that is the exact gap a TEE closes: it guarantees confidentiality for data in use.
That guarantee has a boundary. A TEE proves which code is running through a cryptographic measurement, but that measurement says nothing about whether the code is safe, well-written, or free of bugs. Attestation confirms identity, not intent. This distinction surfaces again once attestation enters the picture directly, because a verifier who trusts a measurement is trusting that the measured code behaves as documented, not that it is flawless. Other approaches exist in principle, too: homomorphic encryption and secure multiparty computation can both keep data encrypted through the entire course of a computation, without ever decrypting it. Both impose enough computational overhead that real-time LLM inference under either scheme remains impractical today. That's why the TEE approach, with its hardware-rooted guarantee and comparatively modest overhead, is the one actually running in production systems.
Two nested TEEs for LLM inference on a GPU
Protecting an LLM request end to end takes a CPU TEE and a GPU TEE working together, not either one alone, because the threat surface runs across both the process orchestrating the model and the accelerator actually doing the matrix multiplication. Why does the CPU alone not suffice, given that it is the part actually running the application code? Because the CPU hands the heavy computation to the GPU, and a GPU boundary that is not separately secured becomes the weak link regardless of what protections surround the CPU.
Earlier CPU-only TEEs, Intel SGX chief among them, had memory limits severe enough to make large-scale machine learning workloads impractical to run inside them at all. NVIDIA's H100 was the first GPU to support a Confidential Computing mode, encrypting and isolating all code and data across the entire chip, streaming multiprocessors and high-bandwidth memory included, with a 2025 survey reporting overhead of 4 to 8 percent for H100 CC.
Stanford's Hazy Research group describes the setup now common in production: an AMD EPYC 4th-generation "Turin" processor running SEV-SNP supplies the CPU-side TEE, and every piece of information passed between CPU and GPU is separately encrypted on top of that, which amounts to two nested TEEs rather than one. The trust relationship between the two is asymmetric by design. Phala's architecture documentation lays out the chain: the GPU goes through a secure measured boot, then opens a secure channel to the CPU TEE using the SPDM protocol, so the GPU authenticates itself to the CPU, but the reverse does not happen, before any model data is allowed to move. Data leaving the CPU's encrypted memory crosses the PCIe bus inside encrypted bounce buffers and is only decrypted once it reaches the GPU's own secure memory, so the host cannot dump GPU memory contents even with physical access to the machine. None of this requires the model code itself to change. From the perspective of PyTorch or an equivalent framework, inference proceeds exactly as it would on an unprotected machine, because the encryption and isolation are enforced transparently by the hardware underneath it.
Remote attestation and verifying what is running before sending anything sensitive
Hardware isolation by itself gives an enclave no way to prove anything to someone outside it. A remote client has no inherent reason to believe a given server is actually running inside a genuine TEE rather than claiming to. Remote attestation is the mechanism that turns that isolation into something a client can check before trusting it with anything.
The platform computes a cryptographic hash of the exact software image loaded into the enclave, and a hardware root of trust signs an attestation document carrying that measurement. A client receiving this document checks the signature against the vendor's certificate chain and compares the measurement against an expected, known-good value. Any modification to the code running in the enclave changes its hash and causes the check to fail. Blaxel's explainer notes that platform certification keys allow organizations to run their own attestation infrastructure, so verification does not have to depend on a centralized vendor service controlling the process.
In a GPU TEE deployment, this produces two separate attestation artifacts rather than one: the CPU TEE's attestation, covering the bootloader, operating system, and application, and the GPU's own attestation, covering its firmware version, secure boot status, and the hash of the loaded GPU program. Phala's documentation states the condition: a client proceeds to trust the enclave only if both attestations check out against known-good measurements. The open-source TrustedGenAi project shows what this looks like in practice. Its attestation endpoint returns a platform field, a tee_verified boolean, and a vm_size, backed by Azure-signed PKCS7 documents and TPM PCR attestation underneath, so a client can check all of this programmatically before a single prompt leaves the client machine. One might argue that this still just moves trust from the operator to the vendor's signing key rather than eliminating it. That is true as far as it goes, but it shifts trust from an operator who could change behavior at any moment to a hardware root of trust fixed at manufacturing time, which is a meaningfully smaller attack surface.
Binding attestation to the encrypted channel
A valid attestation document proves that a genuine enclave is running somewhere. It does not, by itself, prove that the enclave a client is currently talking to is the one that produced that document. An attacker could, in principle, capture a legitimate attestation and replay it over an entirely different connection, presenting someone else's proof as their own. Remote-attestation TLS, or RATLS, is the fix for exactly that gap.
RATLS embeds a freshly generated nonce and the enclave's own transport public key directly into the attestation document, so the proof is tied to this specific channel rather than standing in for any enclave running the same code. Hazy Research's TEECHAT protocol implements this as an ephemeral key exchange: the client and the remote enclave derive a shared secret together, the enclave then produces an attestation proving it is a genuine H100 running in Confidential Computing mode, and every message that follows carries encryption, a signature, and nonce protection. AEGIS applies the identical principle to API routers specifically: the client's TLS session terminates inside the enclave itself, the host machine never holds the interaction in plaintext at any point, and a small client-side component checks the enclave's measurement and refuses to transmit the request body until that check passes.
What makes a design like this auditable is how small the trusted portion of it is kept. The trusted path inside AEGIS runs 851 lines of code, a deliberately narrow surface that can be read in full, audited, and re-measured. That is the TCB minimization principle at work: the smaller the piece of code a client has to trust, the more realistic it is that someone actually checked it.
The prompt's path between client encryption and enclave decryption
With the attested, session-bound channel in place, the prompt's actual journey can be traced step by step. It is encrypted on the client machine before it ever leaves, travels as ciphertext through every router, load balancer, and gateway along the way, and is decrypted for the first time only once it reaches the CPU enclave at the far end. The model never encounters plaintext anywhere outside the boundary of the two nested TEEs.
The encryption applied on the client side uses the shared secret derived during the RATLS handshake described above, and only that handshake's parties hold a key capable of decrypting it, excluding the network carrying the packets, the cloud operator hosting the machine, and the API gateway routing the request. Once the ciphertext reaches the enclave, the CPU TEE decrypts the prompt, tokenizes it, and passes the resulting model inputs across to the GPU TEE over the encrypted PCIe path described earlier. At every stage, plaintext exists only inside hardware-protected memory. Inference then runs entirely within that nested boundary, and the model's response is encrypted and signed before it ever crosses back out of the enclave.
Phala's demonstration in February 2025 is the clearest production example of this whole chain operating at once. A 70-billion parameter distilled version of DeepSeek R1 ran inside a GPU TEE, users queried it with full data confidentiality, and the model's own weights were decrypted only inside the enclave, protecting the user's data and the model owner's intellectual property at the same time. That second point is easy to overlook: the architecture protects the prompt from the operator, and it protects the model from everyone else, including the operator running the hardware. Phala's own documentation makes this explicit, stating that even Phala's network operators see only encrypted data passing through their machines, so users are trusting the hardware's guarantees rather than trusting the company. Anthropic's June 2025 whitepaper on confidential inference backs up the underlying claim at the hardware level: with the right hardware support, TEE memory stays inaccessible even to the operating system kernel, and debugging tools like breakpoints and execution traces are disabled inside the TEE, so there's no way to monitor or alter what's running from outside it. By the time a response reaches the client, the cloud operator has seen nothing but encrypted bytes passing through infrastructure it controls but cannot read.
Applying this architecture to LLM API routers
Everything described so far protects the inference endpoint. But what happens if the model server is fully sealed while the router sitting in front of it is left running in plaintext? The same vulnerability TEEs close at the model server reappears one layer up, and a router holding plaintext is still a man-in-the-middle position capable of rewriting, injecting, or exfiltrating data, regardless of how well-protected the endpoint behind it is.
Client-side defenses aimed at this problem, policy gates, anomaly detection, response logging, all share the same weakness: they observe only the responses that get delivered back, so they offer nothing to protect secrets while the request is still in flight. AEGIS addresses this by confining plaintext handling to a small hardware-enclave component, while authentication, scheduling, and accounting logic stay on the untrusted host without ever touching the actual body of the interaction. This does add cost, though it is a modest one: the local relay overhead AEGIS introduces runs around six milliseconds per request, comparable to the overhead Hazy Research measured for TEECHAT's message encryption.
This extension of the architecture to the routing layer matters most for agentic systems specifically. Coding assistants and autonomous agents increasingly route their traffic through shared API gateways, and under today's standard architecture, the agent extends complete trust to that router. An agent that cannot verify its router is just as exposed as a user who cannot verify the model server behind it, and the fix in both cases is the same attested, session-bound channel rather than a different one.
The performance overhead that remains
GPU TEEs used to carry a reputation for being too slow for real-time inference, a reasonable concern when the technology was new. Current benchmarks on production hardware tell a different story, and that older concern no longer holds up as a reason to avoid the architecture.
Hazy Research's TEECHAT prototype measured roughly 10 milliseconds of added overhead from message encryption and decryption, and once the model is large enough and the inputs long enough, total TEE latency drops below one percent of overall request time. What remains in question is not speed but scope, how completely the attested boundary actually covers the system around it, a harder, more interesting question than whether the architecture runs fast enough.
