Securing the CUDA Driver Stack in Confidential GPU Deployments

Confidential computing on GPUs solves one specific problem: data has to be decrypted to get processed, and that decryption window used to sit wide open to anyone with host access. This piece walks the CUDA driver stack layer by layer, from the kernel driver down to GPU firmware, to show where hardware and cryptography actually close that gap, and where the closed-source parts of the stack still ask practitioners to take something on faith.

Encryption at rest and TLS in transit cover two-thirds of the data lifecycle. The third piece, data in use, sat unprotected for years because there was no practical way to run computation on ciphertext at scale. Confidential computing fixes this with hardware-rooted trusted execution environments (TEEs) that keep data encrypted in memory and expose the keys to nothing outside the enclave, backed by cryptographic attestation instead of a vendor's promise or a compliance checkbox. That matters because regulated industries under GDPR, the EU AI Act, or various sovereign data mandates have faced a binary choice for years: keep sensitive data on-premises and lose access to the largest models, or send it to a cloud API and accept the exposure. Confidential GPU compute is the mechanism narrowing that choice, and Major cloud providers including Microsoft and Google are already running production confidential LLM inference pipelines built on CPU and GPU TEE foundations. This is not a research demo anymore.

So the software underneath that hardware promise actually needs examining layer by layer, to see where the trust has to be earned rather than assumed. The hardware is trustworthy. The closed-source driver code sitting on top of it is not something anyone outside NVIDIA can actually check, and that gap gets glossed over far too often in how this technology gets described.

How the CUDA driver stack is structured and why its layered design creates distinct trust boundaries

Four layers make up the CUDA driver stack, and each one runs at a different privilege level with a different trust status attached to it. At the bottom sits the kernel-mode driver, NVKM, running at ring 0 on the host, the most privileged execution level a CPU offers. Above it sits the user-mode driver, libcuda, which handles most of the actual logic for translating CUDA operations into hardware commands. The Unified Virtual Memory (UVM) driver manages the page mappings shared between host and device memory. And inside the GPU itself, firmware and microcode execute as part of what NVIDIA calls the Trusted Computing Base when confidential computing (CC) mode is active.

Here's the detail that actually matters for anyone trying to reason about trust: NVIDIA open-sourced the kernel driver, but libcuda stayed closed. And libcuda is where most of the operation-handling logic lives. So the translation from a high-level CUDA API call, something like cudaMemcpy or a kernel launch, down to the low-level commands the GPU hardware executes, happens inside a black box. The driver mediates every single processor-to-accelerator interaction, and it does not show its work.

That's a detail with real weight, not a footnote. It's the central design tension of the whole system, and it deserves to be named as such. It means practitioners cannot directly check whether CC-mode protections get enforced correctly at the userspace layer. Trust in that layer has to be indirect, built through attestation reports and researcher instrumentation rather than through reading the source. Each boundary between these four layers is a place where a compromise could undermine guarantees established below it: a bug or backdoor in libcuda could theoretically misrepresent what's happening to NVKM, which could misrepresent state to the firmware, and so on down the chain. Picture the stack as application, then CUDA runtime, then libcuda in userspace, then NVKM in kernel space, then GPU firmware, then the hardware itself. Two of those layers are open for inspection. The rest are not, and the CC trust boundary sits somewhere in that unlit middle, which is exactly the territory the rest of this piece tries to map.

What hardware security engines the H100 Hopper architecture introduced and what they do and do not protect

Hopper, released in 2022 as the H100, was the first GPU architecture to support a hardware TEE. CC workloads became available with CUDA 12.2 Update 1 and reached general availability with CUDA 12.4. Getting there took more than a firmware flag: NVIDIA had to write new secure firmware and microcode, build CC-capable code paths into the CUDA driver, and construct an entire attestation verification flow from scratch.

What Hopper actually added, at the hardware level, breaks into four pieces. The Compute Protected Region (CPR) is a firewall-protected slice of GPU memory where sensitive computation happens. A PCIe firewall blocks the CPU from touching most GPU registers and all CPR memory once CC mode is active at boot. An NVLink firewall does the same job for peer GPUs, so one GPU in a multi-GPU system can't reach into another's protected memory. And the DMA engines are the only user-mode-accessible engines allowed to move data in or out of the CPR at all, with the hardware pre-encrypting everything that leaves using AES-GCM 256.

Here's the limit worth sitting with, because it's the one people get wrong most often: HBM (High Bandwidth Memory) contents are access-controlled, but they are not encrypted at rest. Model weights sitting in HBM during inference are plaintext bits. What protects them is the access-control layer, the firewalls deciding who can read that memory, not encryption of the storage itself. Anyone leaning on CC for model-weight confidentiality needs to understand that "access-controlled" and "encrypted at rest" are two different security properties, and treating them as the same thing is a mistake, not something to soften with a caveat.

Blackwell extends Hopper's CC capabilities and is described as the first TEE-I/O capable GPU in the industry, delivering throughput nearly identical to unencrypted mode. That's a meaningful shift: the historical tax for running confidential rather than unencrypted has shrunk substantially. Vera Rubin, announced for CES 2026 with first systems expected in the second half of 2026 through AWS, Microsoft, Google, and OCI, goes further still. With Hopper and Blackwell, the CC domain stayed confined to the GPU, because the accompanying Grace CPU had no CC support of its own. Vera Rubin brings a CC-capable Vera CPU into the picture, which means the encrypted domain can span CPU, GPU, and NVLink across an entire NVL72 rack. Each generation narrows the performance gap and widens the boundary of what's actually protected. But the HBM at-rest limitation persists across all of them unless a future architecture addresses it directly, and nothing on the current roadmap says it has been.

How the bounce buffer and copy-engine key scheme secure the CPU–GPU data path in practice

CPUs and GPUs are separate processors connected over PCIe, and PCIe is a bus an adversary with host control can watch or tamper with. That's the specific threat the bounce buffer mechanism exists to close. The NVIDIA driver, running inside the CPU TEE, sets up an encrypted intermediary buffer in shared system memory. Every piece of CPU-to-GPU traffic, command buffers, launched kernels, all of it, passes through that buffer encrypted, which shuts down in-band attacks from a host observer sitting on the bus.

None of this requires application code changes. The CUDA driver and GPU firmware handle the encryption on their own, so long as the hardware, drivers, and attestation chain are all configured correctly. There's a performance cost, though it lands in a specific place: accelerator compute throughput and HBM bandwidth stay at parity with non-confidential mode, but the processor-to-accelerator interconnect itself gets bounded by how fast the processor can encrypt, roughly 4 GBytes per second. For workloads that move a lot of data across that link rather than keeping it resident on the GPU, that's the number to plan around, and it's a real bottleneck, not a rounding error.

So what's actually happening inside that closed-source driver during encryption? A paper describing research sometimes called Blueprint, Bootstrap, and Bridge (arXiv:2507.02770) answers that by preloading a customized OpenSSL library and intercepting every cryptographic call libcuda makes. Turns out the CUDA user-mode driver leans on standard OpenSSL APIs for encryption, decryption, and MAC verification, which means researchers could watch, from outside the black box, exactly what the closed-source layer was doing without ever decompiling it.

The driver pulls two categories of keys from the kernel-mode driver through an API called Getkmb. The first category, cpu_sec2_{data,hmac}_user keys, ties to NVIDIA's SEC2 engine and handles encrypted data transfer along with integrity verification. The second, lce{x}_{h2d,d2h}_user keys, attaches to specific logical Copy Engines and handles encrypted transfers in each direction between CPU and GPU. That Getkmb retrieval path is a concrete, observable trust boundary, a place where researchers can actually check that CC protections are being enforced rather than just trusting that they are.

But here's the limit of that visibility: what happens to data once it's inside the GPU's CPR is governed by firmware, and firmware behavior doesn't show up in an intercepted OpenSSL call. The bounce buffer protects the transfer path. It says nothing about what happens after the data arrives.

How the secure boot and bootstrap sequence establishes the CC trust boundary before any workload runs

Reaching a secure, confidential-computing-ready state takes an orchestrated sequence across firmware, driver, and hardware, not a single switch flip. The bootstrap, per the same research (arXiv:2507.02770), runs through four stages. Secure boot first: only firmware components signed and attested by NVIDIA get to run in CC mode. Then key generation, where unique encryption keys get created per guest driver component during initialization. Then firewall establishment, raising the PCIe and NVLink firewalls before any workload gets admitted to run. And finally device attestation, where the GPU driver runs a key-exchange sequence using SPDM messages to authenticate and attest the GPU hardware TEE.

None of that works without two things already in place on the host side. A CPU TEE, either AMD SEV-SNP or Intel TDX, has to be managing confidential VMs (CVMs) securely. And a hypervisor with CC support, KVM being the common example, has to be handling resource allocation in a way that respects that trust boundary.

This is a dependency chain, plain and simple. If any earlier step fails or gets skipped, the later steps can't deliver what they promise: attestation can only verify integrity that was actually established in the first place. Layering all of this new protection on top of hardware and drivers that also need to preserve backward compatibility adds real implementation complexity, and that complexity is itself a documented attack-surface concern in the research (arXiv:2507.02770).

What happens when firmware verification gets skipped or left out entirely? The MOLE attack on Arm Mali GPUs, described at ACM CCS 2025 (doi:10.1145/3719027.3744823) and again in a related paper (arXiv:2510.22566, FAARM), shows the answer concretely. Adversaries with kernel privileges injected malicious firmware, bypassed memory protections and exfiltrated data at over 40 MB/s. This is a documented case, not a hypothetical, and it's a direct precedent for exactly why NVIDIA requires signed firmware only in CC mode, and why that requirement needs checking in deployment rather than being assumed to work.

How attestation verifies the full driver stack and where the verification chain can fail

Attestation exists so a relying party can confirm the GPU and its driver are running legitimate, unmodified software before any sensitive workload gets admitted to run on it. Every H100 carries a unique ECC keypair and public certificate as a hardware identity anchor, and NVIDIA's OCSP service lets a verifier check that certificate's validity and whether the GPU has been revoked from CC eligibility. The attestation report itself follows the DMTF SPDM 1.1 MEASUREMENT response format.

For driver attestation to actually hold, several things have to check out at the same time, according to NVIDIA's attestation documentation. The driver's Reference Integrity Manifest (RIM) schema has to match the expected format. Its certificate chain has to verify. Every certificate in that chain has to pass OCSP validation. And the RIM signature has to verify, with the driver version matching what's actually reported by the GPU. Operationally, this involves pulling driver version, GPU certificates, and the attestation report, and a verification step that parses the report and checks its measurements against golden values stored in the RIM.

Even after all of that passes, an operator still has to explicitly toggle the GPU's ready state to ON before CUDA programs can actually run in CC mode. Attestation doesn't self-activate anything, and that manual step is easy to miss in an automated deployment pipeline.

There's a practical wrinkle for anyone running this in production. As of August 15, 2025, NVIDIA's Attestation Services, meaning the RIM Service, the OCSP Service, and NRAS, all require API key authentication. Unauthenticated requests get rate-limited rather than blocked outright, but any deployment that hasn't updated its integration to include a key is going to see attestation degrade or fail at exactly the moment it matters most.

Multi-GPU systems introduce another failure mode. If even one connected NVSwitch in the topology doesn't support attestation, the attestation SDK raises an error condition, so multi-GPU deployments introduce additional components in the attestation chain beyond the GPUs themselves. NRAS has since stabilized multi-GPU and multi-switch attestation under its /v3 APIs and moved claim versioning to 2.0, so operators still running earlier API versions have a migration to plan for. An alternative path runs through Intel Trust Authority, which can use a locally deployed GPU verifier instead of forwarding evidence directly to NRAS's cloud endpoint, though that local verifier still needs to reach NRAS for RIMs and OCSP collateral. It cuts down real-time dependency on NRAS uptime. It doesn't remove it.

And the same opacity that shows up everywhere else in this stack resurfaces here too. Attestation measures firmware and confirms driver version. It does not measure the internal execution logic of libcuda. It confirms what's running, but it says considerably less about everything that running component actually does once the workload starts.

What the closed-source userspace driver means for independent auditability and how researchers have partially compensated

The structural issue running through this entire piece comes down to one fact: most of CUDA's operation-handling logic lives in libcuda, and libcuda stayed closed even after NVIDIA open-sourced the kernel driver. The translation from a high-level API call to a hardware command remains, by design, opaque to anyone outside NVIDIA. No one forgot to fix it; the oversight was left in place deliberately. It's a deliberate choice, and the entire research effort described in this section exists because of it.

Two research approaches have made real headway into that opacity anyway. One, described in work out of Queen's University (arXiv:2604.26889), takes advantage of the open-sourced kernel driver by instrumenting the memory-mapping path and placing a hardware watchpoint on the userspace mapping of the GPU's doorbell register. That watchpoint catches complete command submissions the moment they get committed, which means researchers can recover the actual hardware command streams the closed-source driver emits, with full integrity, without ever touching libcuda's source directly.

The other, the IBM Research and Ohio State work already referenced above (arXiv:2507.02770), preloads a customized OpenSSL library to intercept every cryptographic call libcuda makes, paired with instrumentation of the kernel-mode and UVM drivers to see how they interact with the userspace layer. Together, these two approaches have surfaced real, concrete behavior: DMA submission modes, how copy-engine keys get retrieved, even differences in command footprint across CUDA versions.

What neither approach fully resolves is the internal decision logic sitting between those observable calls, the actual reasoning libcuda performs that never surfaces as a watchpoint hit or an OpenSSL call. Whether CC-mode enforcement is airtight, or whether some edge case slips past it unnoticed, isn't something either methodology can answer with certainty. The findings from the Blueprint, Bootstrap, and Bridge paper were disclosed to NVIDIA's PSIRT team before publication, which places this research squarely in the tradition of adversarial audit rather than exploit publication for its own sake.

For anyone deploying this stack today, that leaves a real structural gap. Independently verifying CC guarantees at the userspace driver layer means either trusting NVIDIA's attestation measurements outright, or reproducing research-grade instrumentation of the kind described above. There isn't a third option on the table right now, and pretending otherwise doesn't make the gap go away.

How to assess where the security posture holds and where gaps remain across the full stack

Walking back down the stack, the picture that emerges is neither a clean pass nor a clean failure. Hardware protections on Hopper and Blackwell are real and independently verifiable. The PCIe and NVLink firewalls, the CPR, and the AES-GCM 256 encryption on data leaving protected memory are all enforced in silicon, not in a driver that could theoretically be told to lie. Attestation gives a relying party a genuine, cryptographically-backed way to confirm firmware and driver versions before trusting a workload to the GPU at all. On that half of the ledger, the confidence is earned.

What matters is what happens once the hardware does what NVIDIA says it does. It's whether the layers mediating access to that hardware, namely libcuda, get watched closely enough, and the honest answer is: not as closely as the hardware deserves. The gaps concentrate there, in the part of the stack that deserves the most skepticism rather than the silicon. The HBM at-rest plaintext limitation is a known, documented tradeoff, not a secret, but it's one that's easy to miss if a team assumes "confidential computing" means everything touching the GPU is encrypted everywhere, always. It doesn't. It means specific regions and specific transfer paths are protected, and the rest depends on access control holding.

The bounce buffer and copy-engine key scheme close the processor-to-accelerator transfer path convincingly, and researcher instrumentation has made that scheme concretely auditable rather than a black-box claim. The bootstrap sequence, secure boot through firewall establishment through attestation, is a dependency chain where every link has to hold, and the MOLE precedent on Arm Mali shows exactly what a broken link looks like in practice: kernel-privileged firmware injection, bypassed memory protection, sustained exfiltration at real throughput. This risk has already materialized, on a different vendor's silicon, doing the exact thing signed-firmware requirements exist to prevent.

So where does that leave a practitioner trying to assess actual posture rather than marketing language? Trust the hardware firewalls and the attestation chain as far as they've been independently instrumented and verified, which by the research cited here is fairly far. Treat the internal logic of libcuda as an open question, not a settled one, because it structurally is one: nobody outside NVIDIA can read that code, and the the instrumentation method described above, OpenSSL interception, only light up the parts of libcuda's behavior that happen to cross an observable boundary. Everything between those boundaries is inferred, and that distinction should shape how much weight any single deployment puts on the words "confidential computing" without asking which layer is actually doing the protecting.

None of this means confidential GPU computing is a broken promise. Meta, Microsoft, and Google running production inference pipelines on this stack is evidence the hardware foundation holds up well enough to build on. But a coherent security posture means knowing precisely where that foundation ends and where inference, attestation, and researcher goodwill start picking up the slack. That's the throughline connecting every layer covered here: hardware trust gets earned in silicon and verified through attestation, while software trust in the closed layers gets earned through disclosure, instrumentation, and a research community willing to do the adversarial audit work that NVIDIA's own documentation can't fully substitute for.