Multi-GPU Confidential Training Across NVLink and InfiniBand

Multi-GPU confidential training runs into a hard physical fact: the instant data leaves one GPU package and crosses a wire to reach another, anyone tapping that wire can read it. This piece traces how each layer, from a single Hopper chip up to a full Vera Rubin rack, closes that gap, and what each fix costs in speed.

Start with the baseline problem, since it explains why any of this hardware exists. Traditional encryption locks data down at rest and in transit, but the moment a GPU pulls a tensor into memory to run a matmul, that tensor sits there in the clear. On a rented cloud instance, the hypervisor operator has, in principle, a line of sight into that memory: model weights worth months of training compute, or a customer's proprietary dataset, decrypted and sitting in GPU memory for anyone with hypervisor-level access to inspect. NVIDIA's answer is Confidential Computing (CC), a GPU trusted execution environment (TEE) that keeps data encrypted even during active computation. It solves the single-GPU version of this problem well. What it doesn't solve, not on its own, is what happens once a model needs eight GPUs, or seventy-two.

This piece tracks that real shift. Modern large models routinely blow past what a single GPU's HBM can hold, so training runs across many GPUs at once, wired together through NVLink inside a server and InfiniBand between servers. Every one of those wires is a new place data can leak. The threat model anchoring everything below is narrow and specific: only the CPU package and the GPU package are trusted. NVLink, NVSwitch, and PCIe are all assumed hostile, open to snooping or tampering by anyone with physical or privileged access to the fabric. For a cloud where the hardware isn't yours, that's not paranoia. That's the correct starting assumption.

How GPU TEEs are constructed at the hardware level

NVIDIA's H100 brought TEE technology to GPUs commercially, with the full stack coming together through 2023. Before Hopper, confidential computing existed for CPUs (AMD SEV-SNP, Intel TDX) but stopped at the PCIe bus. Hopper pushed the trust boundary onto the GPU itself, giving the chip doing the actual math its own fenced-off memory for the first time, not just the processor feeding it work.

In CC mode, the GPU runs isolated, and every transfer between CPU and GPU gets encrypted instead of passed in plaintext. The mechanism is a bounce buffer: instead of the CPU writing straight into GPU memory, data passes through a staging area protected by hardware AES-256-GCM. Before any of that happens, the two sides need a shared key, and that negotiation runs over SPDM (Security Protocol and Data Model), the standard for setting up session keys between a CPU TEE and a GPU TEE. Secure boot backs this up by making sure only signed, verified firmware ever runs on the card, closing off one of the more obvious ways to inject malicious code. Then remote attestation lets the GPU prove, cryptographically, what firmware and configuration it's running, so a confidential VM (again, SEV-SNP or TDX on the CPU side) can decide whether to trust it and let it into the secure session.

Hardware support keeps widening. H100 started it, and B200 plus the RTX PRO 6000 Blackwell Server Edition carry it forward, spreading from data-center parts into workstation hardware too. For single-GPU inference on large models, this costs almost nothing: benchmarks on single-GPU inference workloads in CC mode land close to native, unprotected performance. The overhead problem that dominates the rest of this piece isn't a single-GPU problem at all, and that's worth sitting with for a second, because it means the fix for Hopper was never going to be "make the chip faster." It shows up specifically once GPUs have to talk to each other, and that's where the real argument of this piece begins.

What NVLink and InfiniBand each do, and why the distinction matters for security

NVLink and InfiniBand solve different problems, and mixing them up is where a lot of confusion about "multi-GPU security" starts. NVLink connects GPUs to each other inside the same server, chips on the same board. InfiniBand (or, in some deployments, Ethernet) connects whole servers to each other across a cluster. One is intra-node, the other inter-node, and a training job at scale leans on both at once.

Sitting above both is NCCL, NVIDIA's Collective Communications Library, which handles the actual choreography: all-reduce, reduce-scatter, all-gather, broadcast, whatever pattern the training job needs to sync gradients or shard activations across however many GPUs are in play. Inside a node, NCCL reaches for NVLink and NVSwitch first, since that's the fastest path, falling back to PCIe peer-to-peer and, as a last resort, host-staged bounce buffers if nothing faster is available. Across nodes, it hands off to InfiniBand through Mellanox HCAs running the mlx5 driver, using GPUDirect RDMA to pull data straight out of GPU HBM without routing it through host memory first. That RDMA path is why InfiniBand still anchors most serious HPC and AI clusters: it's the lowest-latency, highest-throughput way to move data machine to machine.

The bandwidth numbers make the intra-node fabric's advantage obvious. NVLink 4 on Hopper (H100) tops out around 900 GB/s GPU-to-GPU. NVLink 5 on Blackwell pushes an 8-GPU DGX B200 node to roughly 14.4 TB/s aggregate. The GB200 NVL72, which links 72 Blackwell GPUs and 36 Grace CPUs through NVLink 5 and NVSwitch, reaches about 130 TB/s system-wide, effectively behaving as one enormous accelerator rather than 72 separate cards.

This is where the security story gets uncomfortable. Both NVLink and InfiniBand fall on the untrusted side of the threat model laid out earlier, but encrypting them isn't the same job, because the traffic volumes are wildly different: NVLink carries gradient traffic at close to a terabyte a second inside a node, while InfiniBand carries a comparable logical stream between nodes at a fraction of that raw bandwidth. Protecting both costs very different amounts of performance.

And here's the part worth naming: NCCL offers no help at all. It has no built-in authentication or encryption between ranks, because it was built assuming the fabric underneath it could be trusted. In a CC deployment that assumption is false by design, which means every bit of cryptographic protection has to come from the TEE layer sitting underneath NCCL, not from NCCL itself. Anyone deploying confidential training who assumes NCCL "just handles" security has already misread where the protection actually lives, and that misreading is exactly what turns into a surprise 8x slowdown later.

How confidential computing secures data crossing GPU boundaries, and what that costs on Hopper

Because NVLink and PCIe are both untrusted, a GPU TEE can't just hand data to its neighbor and trust the wire. Every pair of GPUs that needs to exchange data has to set up its own symmetric key before training starts, and a GPU still can't reach directly into another GPU's Confidential Protected Region (CPR), the fenced-off memory zone where its secrets live. A small unprotected staging buffer sitting outside the CPR acts as the handoff point instead.

The sequence for every single message runs like this: the sending GPU's TEE encrypts the payload and generates a GMAC authentication tag, both land in the staging buffer, and the receiving GPU's TEE decrypts the payload and checks the tag before trusting any of it. That's four cryptographic operations, minimum, per message, for every send and every receive, all day, for the length of the job.

Distributed data parallel (DDP) training is where this stops being merely annoying and starts being expensive. Backpropagation runs ring all-reduce, a scatter-reduce phase followed by an all-gather phase, generating a steady, high-volume stream of gradient synchronization messages, far more traffic than the occasional transfer between processor and accelerator that earlier CC benchmarks focused on. Add more GPUs to the ring and the number of communication sub-steps grows linearly, and so does the cryptographic overhead riding on top. Asynchronous overlap, the trick that normally lets communication hide behind computation so the GPU never idles waiting on the network, doesn't help much here: when securing and verifying a flood of small messages is the bottleneck, there's no computation left to hide behind it.

The measured numbers, from a study running four GPU TEEs (Characterization of GPU TEE Overheads in Distributed Data Parallel ML Training, arXiv:2501.11771), are stark. Average per-iteration slowdown across the benchmark suite came to 8.68x versus an unsecured DDP baseline. The worst case, training GPT-2-XL, hit 41.64x. Part of that traces back to the encryption throughput itself: NVIDIA's spec sheet claims 4 GB/s for processor-side AES-GCM, but the study's OpenSSL implementation only managed around 1.82 GB/s in practice, a real bottleneck rather than a theoretical one. One mitigation actually moved the needle: enlarging the DDP bucket_cap_mb parameter, which batches more gradients into each transfer and cuts down the number of round trips. That brought GPT-2-XL's slowdown from 41.64x down to 3.03x, a big improvement, but still three times slower than running without CC at all.

There's a compatibility wrinkle on top of that, too. Compatibility constraints in this chip generation's CC mode mean a fair number of standard multi-GPU applications either failed outright or ran degraded, in practice. Fabric-level isolation tools separate tenants from each other in a shared cluster, but none of that substitutes for the per-message encryption the TEE threat model actually demands. Treating those isolation tools as sufficient on Hopper is a mistake worth naming directly: they solve who gets access to the fabric, not whether the bits on the wire are readable. Two different problems, and Hopper only ever answered one of them well.

What Blackwell changes: NVLink encryption and the new multi-GPU security boundary

The single biggest gap on Hopper, according to NVIDIA's own published documentation, was that NVLink and NVSwitch traffic between GPUs went out unencrypted. NVIDIA was explicit that this was deferred, with the fix landing on Blackwell.

With the HGX B200 firmware release, that gap closes. NVIDIA Confidential Computing on Blackwell builds NVLink encryption directly into hardware, alongside fused private signing keys per GPU and remote attestation handled through the NVIDIA Remote Attestation Service (NRAS). Both HGX B200 and HGX B300 now support confidential computing across as many as 8 GPUs with NVLink encryption switched on. Practically, that means the encrypt-then-MAC, decrypt-then-verify cycle described above for Hopper still happens, but now it happens in silicon at the NVLink layer instead of getting bolted on in software running on the CPU.

Published benchmarking on Blackwell puts real numbers to the improvement, with a catch worth sitting with. On a properly patched software stack, single-GPU CC overhead falls to a small fraction of the Hopper-era multiples. Software stack maturity continues to affect results, and the gap between optimized and unoptimized configurations reflects a software problem still being worked out upstream, not a hardware limitation. That gap between patched and unpatched numbers matters more than it might look: it means a good chunk of what gets reported as "Blackwell's CC cost" right now is really just measuring how far along the software stack happens to be, not the silicon.

On the multi-GPU side, overhead tracks the volume of encrypted inter-GPU traffic rather than raw compute, so batching gradients helps but doesn't erase the cost of NVLink encryption. What actually helps is matching the tensor-parallelism degree to what the model needs, since over-provisioning parallelism just generates more encrypted traffic than the job requires. On an HGX B300 running the Qwen 3.5-397B-A17B-FP8 model, turning on CC cost under 8% in throughput and per-token latency across a range of concurrency levels, batch sizes, and token lengths, a very different world from Hopper's double-digit multiples. Published figures suggest HGX B200 retains a substantial performance lead over HGX H200 even with CC fully turned on, though those figures deserve independent confirmation before anyone treats them as settled.

Blackwell also changes who owns what on the fabric. Published research on the B300 HGX platform describes a confidential tenant receiving a partition of a shared NVSwitch fabric, eight GPUs sitting behind a shared NVSwitch fabric, the whole arrangement programmed by a host-side Fabric Manager before the tenant's VM even boots. That's a meaningfully different tenancy model from earlier platforms, where a tenant simply owned a passed-through discrete device outright over a standard bus connection. Sharing a fabric, even a partitioned one, moves the trust boundary and expands exactly what attestation has to vouch for.

How the Vera Rubin NVL72 extends confidential computing to the full rack scale

Vera Rubin NVL72 is designed to run NVIDIA Confidential Computing across the entire system rather than just within a node. NVIDIA calls it third-generation confidential computing, and the pitch is that it holds data security across CPU, GPU, and the full NVLink domain at once: every bus encrypted in transit, not only the link between processor and accelerator that first-generation CC protected.

Concretely, that means the NVLink domain stops being a node-level idea. It now spans the whole rack, and the attestation surface and key-management burden scale up right along with it. Compare the fabric complexity across generations: Earlier HGX nodes ran a smaller set of NVSwitches to serve intra-node traffic; B300 uses two QM3 NVSwitches plus CX-7 management NICs; Vera Rubin pushes the boundary out further still, extending the trusted NVLink domain to rack scale.

For training strategy, that's a real shift. Tensor-parallel and pipeline-parallel schemes that used to treat a node's edge as a hard trust boundary, something to design around carefully, may not need to anymore, at least not at the same granularity. But the physics of the encryption cost doesn't change just because the boundary moved: overhead on cross-GPU traffic still scales with how much traffic actually crosses, whatever hardware generation carries it.

And here's the gap in the record worth being honest about: independent benchmarks of Vera Rubin's CC overhead at full 72-GPU training scale haven't shown up publicly yet. The architectural claim, that every bus is encrypted, is confirmed. What that costs in practice at rack scale is still an open question, and anyone planning around Vera Rubin today is planning around a number nobody has published. Treat any early Vera Rubin marketing number with caution, too, before accepting it as gospel.

Architectural trade-offs a practitioner must reason about when deploying confidential training today

Pull back and the pattern across all four sections holds steady. The dominant cost in confidential multi-GPU training was the coordination and encryption layers wrapped around that compute, the cryptographic work stacked at every package boundary a gradient has to cross during synchronization. That framing hands practitioners three real levers, and the mistake worth naming here is treating them as interchangeable when they aren't.

GPU generation is the first lever, and it's blunt. Hopper-era hardware lacks NVLink encryption, which is exactly the gap behind those 8.68x and 41.64x slowdown figures. Blackwell-era hardware has NVLink encryption built into silicon, and well-configured inference workloads on it show overhead compressed down under 8%. If the choice is between tuning software harder on Hopper or moving the workload to Blackwell, the hardware wins, and no amount of clever batching closes that gap on its own.

Parallelism strategy is the second lever, and it's one practitioners actually control day to day: tensor-parallelism degree sets how much data crosses NVLink on every step, so provisioning more parallelism than the model needs just manufactures extra encrypted traffic and inflates overhead for no benefit. Gradient batching is the third: raising DDP's bucket_cap_mb cuts down the number of encrypt-authenticate-decrypt round trips per iteration, and the Lee et al. results show this narrowing GPT-2-XL's gap from 41.64x down to 3.03x. Real progress, but it's a software patch working around a hardware gap, and it can't close that gap the way NVLink encryption on Blackwell does natively. Anyone reaching for bucket_cap_mb tuning as a substitute for Blackwell hardware is solving the problem at the wrong layer, and the ceiling on that fix is well documented: 3.03x is still 3x slower than uncensored training, not parity.

A fixed cost that doesn't scale with any of this involves setting up peer-to-peer keys for every GPU pair, running SPDM negotiation, and completing remote attestation through NRAS. All of that happens once per session, not once per iteration, so it's a real tax on short jobs and a rounding error on training runs measured in days.

Hardware alone doesn't finish the job, either, and this is the part easiest to overlook. Research on a system called EnclaveX (arXiv:2606.31408, Schambach et al., 2026) points at a gap that CPU and GPU TEEs, however well built, simply don't cover: a Kubernetes administrator can still exec directly into a confidential VM even while both TEEs are fully active. That's an application-layer hole, not a hardware one. So secrets need protecting at the application layer too, with decryption keys released only after attestation succeeds, pulling the orchestration layer itself out of the trusted computing base rather than trusting it by default.

Put the pieces together and a real privacy-preserving deployment, one where user data and model weights genuinely stay out of the cloud provider's reach, needs the whole stack at once: CPU and GPU TEEs, NVLink encryption from Blackwell or later, application-layer key protection, and remote attestation the user controls, not the provider. Each hardware generation so far has closed one specific gap that the previous generation either patched over in software or simply accepted as residual risk. But every generation also opens new tenancy and attestation surface of its own: the shared NVSwitch fabric on B300, the rack-spanning NVLink domain on Vera Rubin. Newer hardware doesn't make that surface safe by default. It has to be reasoned through, generation by generation, the same way the gap on Hopper had to be.