Threat Model for a Confidential LLM Inference Endpoint

A threat model for a confidential LLM inference endpoint has to start lower than most security reviews are comfortable going: at the GPU itself, mid-computation, before any network packet moves. This piece works through that model layer by layer, from the cloud operator who can read VRAM directly, to the co-tenant who needs nothing but timing data, to the adversary who never touches the infrastructure at all, to the one who breaks the trusted environment from inside it. Each layer gets its own adversary assumption and its own architectural answer, because a control that closes one gap says nothing about the rest of them.

Why cloud-hosted LLM inference creates a structural privacy problem

Most production AI workloads now run as cloud-hosted inference, and the GPU compute path underneath them has gone largely unprotected at runtime for most of that history. TLS secures the connection between client and server, and disk encryption protects weights and logs once they're written to storage, but neither does anything for data while it's actually being computed on. GPU VRAM sits unencrypted during that computation, so the one step where the model actually does its work is the one step none of the standard protections touch.

That gap has a specific shape. A compromised hypervisor, or a host administrator with more access than anyone intended to grant them, can read model weights and the prompts being processed directly out of GPU memory. No firewall rule catches this, because nothing at the network or application layer sits between the memory and the party reading it. And what's typically being processed isn't abstract test data: cloud LLM inference regularly handles personal, financial, and health information, which is to say exactly the categories of data that regulators and users treat as most sensitive.

Regulators have started treating that gap as a live compliance question rather than a hypothetical one. Some AI services have tried to address user concerns at the architecture level rather than the policy level: Confidant AI, a privacy-preserving AI assistant built so the service operator itself cannot read user prompts, is one example of a product designed around that constraint from the start. GDPR Data Protection Impact Assessments are now being enforced directly against LLM providers, and Italy's temporary ban on ChatGPT, followed by a €15 million fine over training on personal data without an adequate legal basis, showed what enforcement looks like in practice, even though the Court of Rome annulled that fine in March 2026 on jurisdictional grounds. Quebec's privacy law mandates PIAs with significant penalties, and U.S. FTC oversight is pushing PIA-like reviews for AI services. The sectors most exposed to sensitive data have responded by pulling back: JPMorgan, Citi, and Goldman Sachs have all restricted employee use of LLMs, and healthcare organizations cite HIPAA compliance concerns for the same reason. The restriction exists because confidentiality can't currently be demonstrated.

What confidential computing offers

Confidential computing is the main architectural answer to that gap, and it works by changing what the client has to trust. Instead of trusting the cloud provider's internal policies and promises, the client verifies the hardware's cryptographic state directly through attestation. Trusted Execution Environments run sensitive code inside a hardware-enforced enclave, so it stays confidential and intact even if the operating system around it or the host managing it has been compromised. On the GPU side, VRAM encryption uses a key generated inside the GPU's own security processor and never exposed to anything running on the host, PCIe traffic between CPU and GPU is encrypted so the host only ever sees ciphertext, and remote attestation lets the client confirm the GPU's firmware and security state before sending it anything sensitive. NVIDIA's Hopper generation implements this confidential computing mode with minimal performance overhead, which is a large part of why confidential inference has become something cloud providers can deploy at scale rather than a research curiosity. Intel's 2025 whitepaper on Baidu AI Cloud describes exactly this kind of deployment, using Intel TDX to protect inference workloads at the VM level and hardware-isolate both client data and proprietary model weights.

None of that closes the gap completely, and the limitation is structural rather than a matter of configuration. Attestation happens at boot: TDX and comparable TEEs verify a VM's integrity when it starts up, but they offer no equivalent guarantee once the VM is running. If a component gets compromised after boot, it can still reach VM memory or interfere with a workload already in flight, and attestation alone won't catch it. Both limitations point to the same conclusion: a TEE perimeter is necessary, but it is not sufficient on its own. A real threat model has to name what sits outside that perimeter, and separately, what sits inside it but remains attackable anyway. Later sections work through both, and nearly every mitigation they describe involves a tradeoff between protecting the model, keeping its output unchanged, and keeping it fast, a tradeoff this piece returns to each time a new defense is introduced.

The infrastructure-layer adversary: host and hypervisor access to GPU VRAM

The adversary with the most privileged position is the one sitting inside the cloud infrastructure itself, capable of physical or hypervisor-level access to GPU memory. This is the exact threat confidential computing was built to close, on the condition that the GPU in question is actually enrolled in the confidential computing boundary. The mechanism is concrete: high-bandwidth GPU memory holds its contents for a short window after power loss, and physical access to a node during that window can expose model weights and whatever inference inputs were recently processed, a real concern in colocation facilities and other high-security data center environments. A cloud provider acting in bad faith, or any adversary who has compromised the cloud platform, can try to pull both user prompts and model parameters straight out of VRAM, and none of it trips an application-layer alert.

The adversary assumption here is specific: the cloud operator, the hypervisor, host OS, and platform management layer, is treated as untrusted or already compromised, while the hardware itself is still trusted. That's the assumption VRAM encryption with keys held in the GPU's security processor, PCIe encryption, and remote attestation are built to satisfy, moving trust away from the operator and onto the hardware. What that control doesn't do is protect anything once an adversary is operating from inside the enclave boundary rather than outside it. Side-channel attacks, implementation shortcuts, and supply-chain compromise all remain live threats even when a confidential computing deployment is correctly enrolled, so the sections that follow work through each one.

The co-tenant adversary: how the KV cache became an active attack surface

The KV cache exists purely for speed: it lets inference systems reuse prior attention computations instead of recomputing them, and that reuse creates a timing oracle a co-tenant adversary can exploit to reconstruct someone else's prompt without ever touching the host. Serving systems share the KV cache across users' requests specifically to improve throughput, but sharing it without restriction opens a side channel, because whether a cache hit occurs is observable from outside through response timing.

The research here has moved quickly and concretely. EarlyBird, published by Song et al. in 2024, showed that at a cache block size of 1 (meaning token-level granularity) an adversary can reconstruct a victim's prompt one token at a time, simply by guessing a token and watching for a cache hit, and recover the full prompt this way. PROMPTPEEK, presented by Wu et al. at NDSS in 2025, pushed past that: it recovers full prompts from SGLang's radix-tree cache using fingerprint correlation, without needing to guess exact tokens at all. A paper on efficient KV-cache prompt leakage, published in February 2026, showed that attackers sharing multi-tenant infrastructure can accurately reconstruct confidential queries, including medical symptoms and financial details submitted by other tenants, and that reinforcement-learning-optimized attack models cut the cost of the attack enough to make real-world extraction practical rather than theoretical. A separate survey from Imperial College on confidential computing for agentic AI points to the "Shadow in the Cache" paper from Luo et al., presented at NDSS 2026, which established that storing the KV cache in plaintext is a design defect even when the communication between user and server is fully encrypted. The adversary assumption across all of this is modest: a legitimate co-tenant with no privileged host access, attacking over a normal API using nothing but timing.

The defenses that have come out of this research close some gaps but leave others open. The vLLM V1 engine, as of version 0.26.0, dropped support for block size 1 entirely, which forces an adversary to guess many consecutive tokens correctly before getting any cache-hit signal at all, and brings EarlyBird's practical success rate down to zero on an unmodified, current vLLM deployment. Talaria, described in a February 2026 paper, takes a different approach: its Reversible Masked Outsourcing mechanism cuts token reconstruction accuracy from near-complete recovery down to an average of 1.34%, while still guaranteeing output identical to the unmodified model. KVGov, from 2026, goes further still, offering a single governance mechanism that covers all three known attack families, EarlyBird, PROMPTPEEK, and InputSnatch, under one unified defense, the first mechanism to do so for the whole prefix-cache attack family. The gap that remains isn't in the research, it's in adoption: vLLM's architectural change only protects deployments that actually track and apply upstream updates, and any operator running an older serving stack or a non-default configuration is exposed to exactly the attack the newer default was built to close.

The network-position adversary: timing leakage from autoregressive generation

A step further out from the infrastructure, an adversary who holds nothing more than a position on the network, no host access and no co-tenant account, can still pull sensitive information out of an inference endpoint using timing alone. Autoregressive generation produces exactly one token per forward pass, so a network observer watching response timing can reliably estimate how many tokens came back, and for task-specific models, that output token count correlates with sensitive attributes of the input itself. A related hardware-level variant, presented at USENIX Security 2025 by Gao et al., showed that cache access patterns during the token embedding operation leak token values, and that timing within the autoregressive loop reveals token positions, all without any privileged access to the system.

It deserves treatment as its own category because VRAM encryption and remote attestation don't touch it at all. Encrypting memory and verifying firmware state do nothing to change the timing profile a network observer sees, so this threat stacks directly on top of a correctly deployed TEE rather than being closed by one. You could close it by padding responses to obscure true timing, batching requests so individual token counts aren't observable, or suppressing output token counts outright, but none of those are standard in current serving stacks.

An adversary inside the trusted boundary: implementation shortcuts and supply-chain exposure

The fourth adversary position sits inside the boundary that the first three sections spent their time trying to defend. A correctly enrolled, correctly attested TEE is not automatically secure against what happens inside it, because performance optimizations built into the enclave and third-party code loaded into it can undo the isolation the hardware was supposed to guarantee.

The sharpest illustration of this appeared in 2026, in a widely adopted optimization meant to speed up cryptographic operations in partial TEE-shielded inference setups, using precomputed static secret bases. That optimization introduced a key-reuse vulnerability, and researchers used it to recover secret permutations and the full model weights of a LLaMA-3 model in roughly six minutes, with the approach scaling up to much larger model configurations. The result matters less as a single bug than as a demonstration: a shortcut adopted purely for performance reasons inside a TEE-LLM integration can produce a catastrophic security regression, not a minor one. Imperial College London's research on ARM CCA for on-device model protection backs this up from the architecture side, finding that deploying a model's full weights inside a CCA realm limits an adversary to black-box access and meaningfully mitigates privacy-stealing attacks, while prior work on partial model execution, known as TEE-Shielded DNN Partition, showed that leaving part of the pipeline outside the boundary leaves that part vulnerable. The consistent finding across both is that putting the entire model inside the TEE boundary is the more defensible design, not partitioning it for convenience.

Supply-chain exposure is the second, broader version of the same problem. Modern LLM systems are assembled from pre-trained weights, fine-tuning adapters, datasets, dependency packages, and container images, most of it pulled in through automated pipelines rather than reviewed by hand. A 2026 study looked across five major model-sharing platforms and found tens of thousands of repositories shipping custom model code that runs the moment the model loads. Plenty of users assume that enabling trust_remote_code only grants access to metadata, but it actually executes the full model code, so a malicious artifact placed upstream in that pipeline has an automatic path to execution. The Petridish architecture answers this boundary problem by splitting the service into a per-user prefill process running inside a confidential VM and a shared process that batches attention and generates output, which shrinks the in-TEE attack surface but doesn't fully eliminate the need for per-user isolation. The broader point standing behind both the LLaMA-3 result and the supply-chain findings is that TEE security can be undone from inside the boundary, not only attacked from outside it, which is a failure mode that attestation, by design, has no way to see.

How the serving stack leaks prompts during local inference

Running inference entirely on-device removes the cloud operator from the threat model, since there's no hypervisor, no co-tenant, and no cloud platform administrator in the picture. It does not remove exposure, because the local serving stack still has a full software lifecycle of its own, and prompt data moves through every stage of it. The four adversary positions worked through above were all built around the assumptions of a cloud deployment: an untrusted operator, a co-tenant with API access, a network observer, and an attacker working from inside a TEE. None of those map directly onto a laptop or an on-premises server running its own inference stack, so this case needs its own accounting rather than an assumption that local deployment settles the question.

The lesson across all five adversary positions points in one direction: confidentiality is a property of how the inference architecture is built, determined by where the inference happens to run. A cloud deployment with GPU VRAM encryption, PCIe encryption, attestation, a hardened KV cache, timing-aware response handling, and a vetted supply chain can close more of the threat surface than a local deployment running an unexamined serving stack with default settings. Moving inference off the cloud removes one adversary from the model. It does not, on its own, answer for the rest.