NVIDIA H100 Confidential Computing Mode Architecture
NVIDIA's H100 GPU now encrypts AI model weights in memory against even physically present attackers.
When I first started working with GPU infrastructure in cloud environments, the threat model felt abstract. You had your TLS, your disk encryption, your VPC boundaries. The perimeter felt real. Then I watched a demonstration, in a lab setting, of a privileged process reading model weights directly out of GPU memory on a shared host. No exploit. No zero-day. Just a process with sufficient privileges and direct access to VRAM. That moment clarified something important: the security boundaries the industry had spent a decade refining stopped at the PCIe slot.
The NVIDIA H100, in its confidential computing mode, is the industry's most serious answer to that gap. Understanding why that answer looks the way it does requires working through what the problem actually is, not what it sounds like.
What Confidential Computing Formally Requires, and What the H100 Must Satisfy
The Confidential Computing Consortium, an industry body hosted under the Linux Foundation, defines confidential computing as the protection of data in use by performing computation in a hardware-based, attested trusted execution environment (TEE). Three properties flow from this definition with some precision: confidentiality, meaning code and data cannot be read by unauthorized parties during execution; integrity, meaning neither can be silently modified; and attestation, meaning a remote party can cryptographically verify that the environment is genuine and uncompromised.
These are not aspirational goals. They are design requirements. Each either holds at the hardware level or it does not hold at all.
Before H100, confidential computing on CPUs had matured considerably. Intel TDX and AMD SEV-SNP can establish hardware-isolated execution environments on the CPU, protecting memory from even a compromised hypervisor or host OS. That is meaningful protection for a large class of workloads. But AI inference and training are not that class. The arithmetic throughput those workloads require lives on the GPU, not the CPU. A model running inside an AMD SEV-SNP confidential VM still ships its weights and activations across the PCIe bus into GPU VRAM, where they sit in plaintext, visible to any sufficiently privileged process on the host. No TLS session, no disk encryption scheme, no network firewall addresses that. The CPU TEE is protecting a staging area, not the computation itself.
NVIDIA's stated design goals for H100 CC mode address this directly: confidentiality and integrity of code and data in use, combined with protection against physical attacks, including interposer attacks on PCIe and DDR buses. The threat model is explicit and notably aggressive. The architecture assumes an adversary who may have physical access to the host, may control boot firmware, system management mode, the host OS, the hypervisor, and peripheral devices, and may be able to read host system memory. In other words, the architecture treats full compromise of everything outside the TEE boundary as the baseline threat, not an edge case.
The sections that follow can be read as answering each of the three CCC requirements in turn. Confidentiality through hardware-enforced memory isolation. Integrity through verified boot and locked firmware. Attestation through a cryptographic chain anchored to the chip itself.
The Hardware Root of Trust Burned into Every H100 at Manufacture
A root of trust is only useful if it cannot be forged. Every other security property in the architecture depends on this one.
Every H100 ships from the fab with a unique elliptic curve cryptography keypair fused into the chip. The private key is burned into one-time-programmable fuses, making it immutable after manufacture. NVIDIA's certificate authority issues a certificate covering the corresponding public key, and NVIDIA operates an OCSP service so that any verifier, anywhere, can check whether a specific GPU's certificate is valid and has not been revoked.
The chip also contains a dedicated hardware block called the Confidential Computing Engine, integrated directly into the H100 die and distinct from the tensor core arrays and other compute resources. The CCE is the enforcement point for CC protections; it is not implemented in firmware or driver code that a privileged process could overwrite.
The structural analogy here is to TPM chips in enterprise PCs or to secure enclaves in mobile SoCs. Both architectures place trust in something that exists below the software stack and cannot be spoofed by anything running above it. What H100 does differently is implement this pattern on a datacenter GPU die, for the first time, at the scale required for production AI workloads.
That last point deserves emphasis. Prior to H100, no GPU had hardware-rooted confidential computing capabilities. The per-device fused key is what makes the attestation chain in later sections meaningful rather than circular.
How the H100 Boots into Confidential Mode and Locks Its Own Firmware
CC mode is not the default GPU operating state. It must be explicitly enabled by the host, persistently, before a GPU reset. That reset triggers a defined initialization sequence: the GPU firmware scrubs all GPU state and memory, eliminating any data from a previous tenant; the firmware configures a hardware firewall blocking unauthorized access; and only then does PCIe become active, making the GPU available to the confidential virtual machine.
On power-on, the GPU verifies its own firmware against NVIDIA's digital signature. If the signature is valid, firmware is loaded and then locked against runtime modification. The result of this process is a set of cryptographic measurements, hashes of firmware and configuration state, that are later included in the attestation report. The boot sequence is, in effect, a measurement process as much as an initialization process.
Three operating modes exist. CC-Off is standard GPU operation with no CC features active. CC-On activates all protections and simultaneously disables all performance counters; disabling the counters eliminates a class of side-channel attack surface where timing information can leak secrets. CC-DevTools is a partial mode matching CC-On workflows but with security protections disabled and performance counters re-enabled, intended solely for profiling and development. CC-DevTools must never be used in production. The distinction between these modes is worth internalizing because the performance overhead discussion later in this piece only applies to CC-On, and because anyone who deploys CC-DevTools in a production workload has negated every security property described here.
How the CPU TEE and GPU TEE Are Joined into a Single Trust Boundary
The H100's GPU TEE does not stand alone. It requires a confidential virtual machine on the CPU side as its foundation, established via AMD SEV-SNP or Intel TDX depending on the host platform. The GPU is brought into the CVM's trust boundary through a verified process, not assumed into it.
For inclusion in that boundary, the GPU must authenticate itself as a genuine H100, demonstrate that its certificate has not been revoked, and provide evidence of being in a known good state through its attestation report. All three steps must succeed. A GPU that passes authentication but whose certificate has been revoked does not join the boundary.
Google Cloud's A3 Confidential VMs illustrate this architecture in practice: the CPU hardware prevents direct GPU access to the CVM's private memory, maintaining a strong boundary even between the GPU and the host CPU memory space. The combined result extends the TEE concept across the full compute path. CPU registers, CPU memory, the PCIe bus, and GPU VRAM are all within the protected perimeter when CC mode is active. Each segment of that path is now part of a single trust boundary rather than a series of separately secured hops.
This matters because the attack I described at the opening of this piece lives in the gap between those hops.
The Compute Protected Region and How VRAM Is Partitioned at the Hardware Level
GPU device memory in CC mode is partitioned into two logical regions: the Compute Protected Region and unprotected memory. Up to approximately 90 percent of GPU memory can be designated as the CPR.
The hardware firewall enforces the CPR boundary in a specific way. Direct ingress and egress to the CPR are blocked. The hypervisor and host OS cannot read from or write to it. All data exchange between the CVM and the GPU must pass through controlled, encrypted interfaces. There is no software bypass, because the enforcement is in hardware, not in a driver that a compromised hypervisor could patch.
Inside the CPR, the GPU processes data at full speed in its high-bandwidth memory. There is no per-operation encryption overhead for computation happening within the protected region. Model weights loaded into the CPR are inaccessible to the host for the duration of the session. This directly closes the attack I described at the opening: a privileged host process looking at VRAM sees only the unprotected region.
The CPR's purpose is straightforward. The question was never whether NVIDIA could build fast encrypted compute; the question was whether they could eliminate the plaintext exposure window in VRAM. The CPR is the answer to that question.
How Data Moves Securely Between CPU Memory and the GPU's Protected Region
Closing the VRAM exposure window creates an adjacent problem: data has to get into the CPR from somewhere, and that transfer path is itself an attack surface.
The H100's DMA engine enforces a rule: any data written outside the CPR is pre-encrypted by hardware before leaving the protected boundary. The encryption standard is AES-GCM 256, implemented in the DMA engine itself. This applies to data moving in both directions.
The mechanism that makes this work is a bounce buffer pattern. A shared staging buffer exists in memory accessible to both the CVM and the GPU. Data from the CVM's private memory or from the GPU's CPR must be encrypted with a negotiated session key before placement in this staging buffer. On retrieval, it is decrypted with the same key. Plaintext never exists in the shared region. The session key is negotiated using SPDM, the Security Protocols and Data Models protocol, which is also used for device authentication earlier in the setup process. The GPU driver initiates the SPDM key-exchange sequence, and the session key is scoped to the current confidential session.
This design reflects a pragmatic constraint. A fully isolated system where no data ever crosses a boundary is useless for computation. The bounce buffer pattern accepts that data must cross the CPU-GPU boundary while ensuring that what crosses is never plaintext in any location that an adversary could read.
Remote Attestation: How a User Verifies the GPU Environment Before Trusting It with Data
Attestation is the property that distinguishes confidential computing from security through obscurity. Without it, a user has only the cloud provider's word that CC mode is active and unmodified. With it, the user can verify independently.
The attestation report is a cryptographically signed document produced by the GPU. It contains measurements of firmware, configuration, and boot state, compiled during the initialization sequence described earlier. Critically, it is signed with the GPU's per-device private key, the one burned into fuses at manufacture. Only a genuine H100 with that specific hardware identity can produce a valid signature over that specific set of measurements.
The verification flow is sequential and each step depends on the previous one. A user queries the GPU's attestation report and certificate, then checks the certificate against NVIDIA's OCSP service to confirm validity and non-revocation. NVIDIA's Remote Attestation Service stores a nonce alongside the device certificate hash with a 24-hour time-to-live, which prevents replay attacks using previously captured attestation reports. If attestation succeeds, the user sets the GPU ready state to ON; only then can CUDA programs execute in CC mode.
The chain runs from hardware root of trust through measured boot through SPDM session establishment through the signed attestation report through the OCSP revocation check to the user's trust decision. No step can be forged without access to the fused private key. A compromised host cannot produce a valid attestation report for a modified GPU state, because it cannot forge a signature it does not hold the key to produce.
That is the guarantee. It is worth sitting with the precision of it.
What Existing CUDA Applications Have to Change to Run in CC Mode
The primary design goal for CC mode compatibility was lift-and-shift: CUDA applications should run unchanged. This matters commercially. Requiring rewrites would have been a barrier to adoption that most regulated-industry customers would not clear.
In practice, the software requirements are environmental rather than programmatic. As of the general availability release in April 2024, running in CC mode requires NVIDIA's CUDA 12.4 Data Center Driver at minimum. Deployment is supported in traditional VMs with CPU TEE, and in Kubernetes environments using Kata containers launching confidential containers in microVMs.
The timeline between hardware and software availability is worth noting, not as criticism but as context. H100 launched in late 2022. The CUDA Toolkit 12.2 update in July 2023 added software support for CC. General availability came in April 2024. The hardware capability preceded the production software stack by roughly 18 months. Evaluators who examined H100 in 2023 and concluded CC mode was not yet ready for production deployment were correct at the time; the situation changed by mid-2024.
One operational constraint deserves explicit mention. In CC-On mode, all performance counters are disabled. Standard GPU profiling tools do not function. Development and optimization work must be done in CC-DevTools mode and then validated in CC-On, accepting that the profiling environment and the production environment are not identical in their security posture.
The Performance Cost of Running in CC Mode and Where It Concentrates
Compute inside the CPR runs at full GPU speed. The overhead is not in the arithmetic; it is in the data transfer path. The encrypted bounce buffer that secures CPU-to-GPU data movement imposes latency, and that latency has a relationship with workload characteristics that determines how much it matters in practice.
A 2024 benchmarking study (arXiv:2409.03992) examined this relationship across several LLM sizes. For most typical LLM inference queries, overhead stayed below 5 percent. Larger models approached zero overhead. The pattern is intuitive: as the ratio of compute to input data grows, the encrypted transfer cost amortizes across more floating-point operations, and the overhead fraction shrinks. A small model spends relatively more time moving data and relatively less time computing; a large model inverts that ratio.
High concurrency complicates this picture. A 2026 benchmarking study (arXiv:2607.19353) examining Intel TDX with H100 found that under fixed request-rate conditions, CC mode increased average time-to-first-token by 21.8 percent for Mistral-7B and 27.8 percent for Qwen3-30B-A3B, with global token throughput dropping 17.7 percent and 21.1 percent respectively. The overhead is workload-dependent and concurrency-dependent, not a fixed tax.
The practical implication for deployment planning is that large production inference workloads with substantial model weights, the kind that regulated industries are most likely to want to protect, are also the kind that see the smallest relative overhead from CC mode. Smaller models under high concurrent load show measurable cost. Deployment sizing should account for both dimensions independently rather than assuming a single overhead estimate applies across the workload envelope.
What the H100's CC architecture represents, taken together, is the first serious hardware attempt to extend trust from the CPU to the accelerator across the full computation path. Whether the overhead profile, the software maturity, and the deployment complexity are acceptable for a given workload is an engineering question each team has to answer for itself. But the question of whether the threat model is real, whether VRAM exposure on shared GPU infrastructure is an actual risk for sensitive AI workloads, was settled for me in that lab. The architecture described here is NVIDIA's answer to that threat, built into the silicon.
