Model Weight Protection Inside Confidential Inference Servers
Hardware enclaves protect model weights where policy and access controls cannot.
Model weight theft is a data-in-use problem, distinct from data-at-rest or data-in-transit problems, and that distinction is the whole reason confidential inference servers exist. Weights are the distilled output of years of compute, curated data, and research decisions that competitors and adversaries would rather steal than reproduce. Whoever holds the weights holds the model: they can run it unrestricted, strip out its safety tuning, or push it toward purposes the original developer never intended. Confidential inference servers close one specific gap in that story, the moment a model is loaded into memory and actively running, by building hardware enclaves that even a cloud provider's own system administrators cannot see into.
The cost asymmetry is what makes this urgent. Training a frontier model can run hundreds of millions of dollars in compute alone; running inference on a stolen copy of those weights costs next to nothing. Once weights get out, the attacker's marginal cost drops toward zero while the defender's sunk cost stays exactly where it was. Cloud security has spent most of its energy on data at rest, meaning encrypted storage, and data in transit, meaning TLS and VPNs, leaving the third state mostly unguarded. Policy and access controls have not closed that gap, and there's a reasonable argument they cannot: no amount of contractual language stops a privileged administrator from reading plaintext off a chip. Hardware offers a more durable answer, and working through why took setting the storage and transit cases side by side against this one, because on the surface all three look like the same encryption problem.
The threat landscape RAND catalogued and what it implies for inference servers
RAND Corporation's May 2024 report on securing AI model weights is the most thorough public accounting of how weights actually walk out the door, and it reframes what "security" even needs to mean here. RAND counted 38 distinct attack vectors across nine categories, running from social engineering and insider compromise up to scenarios as extreme as a military seizure of infrastructure. Sit with that for a second: most of those 38 vectors aren't hypothetical. They've already been used, somewhere, by someone.
The adversaries RAND catalogues span multiple types. Nation-states want model weights for strategic advantage; commercial competitors want them to erase years of R&D spend in a single theft; criminal actors want them because a capable stolen model is a product they can resell or run for profit. RAND's benchmarking spells out the resourcing gap in plain numbers: an amateur attacker has under a 20% chance of successfully exploiting a given vulnerability in the machine learning stack, while a well-resourced nation-state clears an 80% chance against that same vulnerability.
That gap is what shapes how inference servers actually get built. RAND's recommendations run to nearly 170 distinct security measures, and for frontier models the guidance pushes toward the top protection tiers precisely because the threat model includes state-level capability. Once a nation-state is on the threat model, software defenses and policy defenses, meaning contracts, access agreements, review boards, carry real limits on their own. Sitting with that 80% figure long enough is what surfaces the actual point: if the attacker can plausibly be a state, the defense benefits from hardware enforcement rather than resting on trust in an organization's good behavior. An 80% success rate against a well-funded adversary is not a number a strict access policy talks its way around.
How inference servers expose weights in ways perimeter security cannot prevent
Here's the mechanical fact underneath all of it: for a model to run, its weights have to sit in memory in plaintext. No version of conventional inference architecture gets around this. Once that plaintext copy exists, anyone with privileged access to the machine, an OS administrator, a hypervisor operator, cloud staff doing routine maintenance, can in principle read it straight off the chip.
The ML tooling around the inference server widens that exposure further. Experiment tracking platforms, model registries, GPU orchestration layers, and open-source model hubs have all shipped documented vulnerabilities that let an attacker run remote code or pull a model out directly. Every extra tool in the pipeline is another door left unlocked.
The less obvious leak sits in the output channel itself. Inference servers today generate something on the order of a terabyte of tokens a day, and that output text is the one channel a provider can't lock down without breaking the product; the whole point of a chat interface or an API is to produce and return text. A November 2025 paper by Rinberg et al. demonstrated steganographic exfiltration: an attacker shapes an LLM's outputs so the tokens themselves encode fragments of the weight data, hidden inside what reads as ordinary generated text. A separate 2025 paper described a wireless covert-channel attack against inference hardware, pulling model parameters out through wireless transmission patterns.
Laying those two 2025 papers next to each other is what makes the harder point visible: in a genuine zero-trust framing, the adversary includes the cloud operator itself. This isn't premised on cloud providers being assumed malicious. The security model has to hold even if one privileged employee, or one compromised admin credential, decides to take a look, and access control offers little help here because it assumes the administrator is trusted by definition. That assumption is exactly what this approach sets aside.
What a Trusted Execution Environment does to the weight-access problem
A Trusted Execution Environment moves the boundary of trust from the organization to the chip. A TEE uses hardware-level isolation to encrypt a region of memory and wall it off from the host operating system, the hypervisor, and any privileged software sitting above it. A cloud provider with full root access to the physical server still can't read what's happening inside that encrypted region. The protection is architectural. Nothing about it depends on a policy anyone agreed to follow.
Walk through the actual sequence, because the order of operations is the whole argument. First, the enclave spins up, and its integrity gets verified through remote attestation, a cryptographic proof that the enclave is running exactly the code it claims, unmodified. Only after that attestation succeeds does the system authorize pulling in the encrypted model image. The model provider's private key then moves into the enclave, and only inside that hardware-encrypted memory does decryption happen. At no point does plaintext touch the host OS, the hypervisor, or an administrator's screen.
Without a TEE, weights loaded for inference sit in plaintext, visible to anyone with host access. Inside one, they stay encrypted end to end, with decryption confined entirely to the hardware boundary. Attestation is the hinge the whole thing turns on: it's what lets a model provider verify, remotely, that the correct and untampered enclave is running before any decryption key gets released. No attestation, no decryption, full stop. That's the mechanism that delivers the guarantee this piece opened with: even a privileged administrator can't get at the weights while they run.
Why CPU-only TEEs are insufficient when the GPU holds the weights
Tracing the actual compute path is what exposes where a lot of otherwise reasonable confidential computing setups fall apart. For LLM inference, the CPU is mostly a traffic director. The actual computation, and the actual weights, live in GPU memory. A CPU-only TEE, however well built, leaves that GPU memory completely exposed. An attacker who can read GPU memory pulls the weights straight out, even while the CPU's enclave sits there fully intact and doing its job correctly.
Following the data itself is what surfaces the detail buried under a lot of confidential-computing marketing: cloud providers selling CPU-level confidential computing instances, built on Intel TDX or AMD SEV-SNP, offer only partial protection for model weights during inference if that's where the protection stops. Data gets decrypted before it crosses over to the GPU, which makes the PCIe bus and the GPU's own memory the real point of exposure. Picture a vault with a steel door and an open window around back; that's a CPU-only TEE guarding an LLM. Selling that as complete protection for inference workloads is, at minimum, an incomplete claim, and providers who market CPU-only confidential computing for LLM workloads without saying so plainly are letting customers assume a guarantee that isn't there.
Closing that gap means extending the TEE boundary onto the accelerator itself: encrypting weights and intermediate computations on the GPU, protecting the data path across the PCIe bus, and extending hardware attestation to cover the GPU, not just the CPU. The resulting architecture has the CPU TEE virtual machine spin up first and stay the trust root, but it then controls and verifies a separate GPU TEE, extending that boundary rather than stopping at it.
How NVIDIA's GPU confidential computing architecture implements the enclave at the accelerator
NVIDIA's H100, built on the Hopper architecture, was the first GPU to support confidential computing. The feature shipped in July 2023 and reached general availability in 2024, which, for a hardware security feature, moved fast.
The mechanism is called the Compute Protected Region, or CPR, and it's worth picturing concretely rather than treating as an abstraction. The CPR is a walled-off chunk of the GPU's own memory, blocked by hardware firewalls from unauthorized access by the host OS or cloud administrators, no matter their privilege level. Data headed for the GPU, including the weights, gets encrypted by the CPU TEE before it ever crosses the PCIe bus. Decryption and loading into the CPR only happens once the GPU's secure components confirm the environment is legitimate. The CPU-to-GPU transfer runs AES-256 encryption, on-die.
Blackwell, announced in March 2024, extends this further by encrypting traffic between GPUs working together on a single large model. That closes an attack surface Hopper left partly open: a model too big for one GPU has to shard across several, and the links between those GPUs were, until Blackwell, a softer target than the CPR itself.
For a sense of what production deployment at this scale actually looks like: Among production deployments, Apple runs its Apple Intelligence workloads through Private Cloud Compute, one example of how confidential inference is being applied at scale. Intel and AMD supply CPU-side TEEs. NVIDIA is the vendor that pushed the enclave concept onto the accelerator, and that matters because the accelerator, not the CPU, is where LLM inference actually happens.
The performance cost of running inference inside a hardware enclave
What does all this encryption cost in speed? A security feature nobody can afford to turn on isn't really a security feature. A 2024 benchmark study, arXiv:2409.03992, measured overall performance overhead on NVIDIA H100 for typical LLM queries at below 7%. That number alone would be a fair trade for most deployments. What's underneath it is more interesting, and it cuts against the intuition most people bring to this.
Overhead shrinks as the model gets bigger, which runs opposite to the instinct that more computation should mean more overhead. The smallest model tested, Llama-3.1-8B, carried the highest overhead of the group. A mid-size model, Phi-3-14B-128k, saw roughly two-thirds of that overhead. The largest model tested, Llama-3.1-70B, saw overhead drop to near nothing on the H100. Working out why a bigger model, protecting more memory, would cost less in overhead means separating out where the overhead actually lives: almost entirely in latency at the start of a request, specifically Time to First Token, and that cost is dominated by I/O and setup rather than the matrix computation happening inside the enclave. Larger models spend more of their total inference time in that computation phase, which barely feels the enclave at all, and that dilutes the fixed setup cost as a share of the total.
Inter-token latency, the pace tokens get generated once a response starts flowing, is barely touched by the enclave. For any application producing longer outputs, the user-facing impact rounds down close to zero. NVIDIA's own vendor-reported figures for H100 and H200 Confidential Computing put performance at 95 to 99% of native, unprotected throughput. The old assumption that security and performance trade off against each other holds only weakly here, at least not at the model sizes where weight theft is worth worrying about in the first place. That's a genuinely counterintuitive result, and it's the reason GPU confidential computing went from research curiosity to production feature in under two years.
Why cryptographic alternatives to TEEs (homomorphic encryption and MPC) haven't displaced them for inference
TEEs aren't the only approach on the table, and it's worth asking honestly why the alternatives haven't won, given that a TEE means trusting a chip vendor's hardware rather than pure math. Three families of approaches exist for protecting weights and inference data: modifying the model itself through watermarking or authentication schemes, cryptographic methods that never decrypt data at all, and hardware-enforced enclaves.
Homomorphic Encryption computes directly on encrypted data, with no trusted hardware anywhere in the pipeline. In theory it's the strongest confidentiality guarantee of the three, since the data is never in plaintext at any point, not even inside a chip. In practice, Fully Homomorphic Encryption runs 100 to 1000 times slower than plaintext computation for complex operations, and the bootstrapping step FHE requires multiplies the cost of every single forward pass through a network. Generative LLMs at production scale simply don't run under current FHE, and treating it as a near-term substitute for TEEs is wishful thinking dressed up as rigor.
Multi-Party Computation lets several parties jointly compute a result without any one of them seeing the others' inputs. The blocker there is communication overhead between parties, which makes MPC too slow for the low-latency, high-throughput demands inference workloads actually have.
TEEs carry their own honest limitation, worth stating plainly instead of glossing past it: the security guarantee a TEE provides is generally weaker than what HE or MPC promise in theory, because TEE implementations can carry hardware bugs, software bugs, or side-channel leaks that a purely cryptographic scheme wouldn't be exposed to at all. A 2025 analysis, arXiv:2509.18886, works through this trade-off directly and lands on TEEs as the only viable method for protecting LLM inference at scale today. That conclusion rests on their balance of security, speed, and generalizability, imperfect as that balance is, rather than on any claim of flawlessness. TEEs win by elimination here, not by being the theoretically superior tool, and that is a distinction worth holding onto rather than smoothing over.
Differential privacy solves an adjacent but different problem, adding statistical noise to protect individual data points, and it offers little confidentiality against a hypervisor that's simply decided to look at raw memory. Hybrid schemes, protecting only the most sensitive layers of a model inside a TEE while running the rest unprotected, do exist for edge devices where a full GPU TEE isn't available. These read as engineering compromises forced by hardware limits, made by teams working within the constraints of an enclave they don't have full access to, not as evidence that partial protection is an acceptable design target when a full GPU TEE is available.
What the enclave boundary means for model providers deploying on third-party infrastructure
What actually changes for a model provider running inference on someone else's cloud? The trust relationship moves from organizational to cryptographic. Instead of trusting a provider's staff and internal policy, the model owner trusts an attestation result. Decryption keys get released only to an enclave that has proven, cryptographically, it's running the exact unmodified environment the provider expects. If that environment has been altered in any way, attestation fails and the keys never leave the provider's control. Admin access on the cloud side carries little weight against that failure, because releasing the key doesn't depend on anyone's judgment call.
That bears directly on user-facing privacy claims too. A user sending a request to a confidential inference server can, in principle, get a cryptographic proof that both their input and the weights processing it were shielded from the infrastructure operator the entire time. A privacy policy is a promise an organization makes about its own future behavior; an enclave adds a technical constraint that holds regardless of what anyone at that organization decides to do later. That difference is not a matter of degree, and providers who blur the two in their marketing are overstating what a policy alone can guarantee. Platforms built around GPU confidential computing have started marketing explicitly toward this, positioning hardware attestation as an answer to the specific risk of running sensitive models on infrastructure someone else physically controls; the ones worth trusting are the ones that publish attestation results rather than just asserting the enclave exists.
None of this closes the RAND taxonomy in full, so it's worth being honest about where the boundary actually sits. Social engineering, supply chain compromise, and physical access to hardware all sit outside what a TEE directly addresses; an attacker who talks an employee into handing over credentials has bypassed the enclave problem entirely by going around it, not through it. What the enclave does close is the specific, structural gap conventional cloud inference struggles with: the moment weights sit decrypted in memory, ready to run, is no longer a moment when a privileged insider or a compromised admin credential gets a free look.
Providers still have real operational work ahead: deciding who holds decryption keys and under what conditions those keys get released, building attestation infrastructure that verifies and logs results reliably, and handling the multi-GPU case that architectures like Blackwell are only beginning to close. But the runtime access gap itself, the one that made data in use the least defended state a model's weights ever pass through, finally has a hardware answer to sit alongside the policy one.
