Confidential Federated Learning With GPU TEEs

Federated learning was supposed to fix machine learning's privacy problem by keeping raw data on the device and sending only model updates to a central server. That promise leaks more than it lets on: the gradients themselves carry information back out, and the server doing the aggregating sees more than most designs let on. This piece walks through why that gap exists, why the usual cryptographic patches only close part of it, and how GPU trusted execution environments (TEEs) are starting to close the rest by pushing hardware-enforced confidentiality all the way down to the compute layer.

Start with the mechanics. In federated learning (FL), a client trains a model locally on its own data, then sends the resulting gradient, not the data, to a central aggregator. The aggregator averages updates from many clients and pushes the new global model back out. No raw record supposedly ever leaves the device, or the hospital, bank, or telecom silo where it lives. That was the pitch, and for a while it got treated as close enough to a privacy guarantee to build products on.

Except the update itself is not neutral. A growing body of research on gradient inversion attacks (GIAs) shows that an adversary with access to a client's gradient can reconstruct the training data that produced it, sometimes with striking fidelity in image tasks. Membership inference attacks go after a narrower question, whether a specific record was part of a client's training set at all, by studying how updates shift over training rounds. Then there's the aggregator problem, less exotic but arguably more consequential: even a well-behaved, non-malicious central server sees every individual client's update before any of it gets averaged into anything. Each one of those updates gives away a lot about the client that produced it.

A second threat runs in parallel, and it gets less attention than gradient leakage even though it matters just as much day to day. FL servers have to trust that each client actually trained honestly on real data. Catching a poisoned or manipulated update is hard, because non-IID data (the norm in FL, since every client's data distribution looks different) already produces updates that look irregular on their own. An adversarial update hides in that noise easily. Defenses that try to catch bad updates by studying the data statistically tend to throw a lot of false positives, flagging honest clients with unusual data as attackers.

So there are really two attack surfaces, not one: the gradient channel running from client to server, and the aggregation environment sitting at the server itself. Most production FL deployments talk a lot about the first and barely acknowledge the second.

Why the standard privacy add-ons don't fully solve either problem

Homomorphic encryption (HE) lets a server compute on encrypted gradients without ever decrypting them, which sounds like exactly what's needed here. In practice, encrypting model weights inflates their size by a factor of 20 to 200 times. For any model with more than a handful of parameters, that communication overhead alone rules HE out of most production FL pipelines. The math holds up fine. The bandwidth bill doesn't.

Differential privacy (DP) takes a different approach: add calibrated statistical noise to gradients before they leave the client, so no single record can be reliably pulled back out of the output. This does hide membership, but it costs model accuracy by construction, not by sloppy implementation. That tradeoff is baked into the definition of differential privacy itself: turn the noise down to protect accuracy, and the privacy protection goes down right along with it. There's no tuning knob that escapes this. It's a real tradeoff, not an engineering bug waiting for a patch.

Secure multi-party computation splits computation across parties so no single one sees the full picture, but the number of messages passing between parties balloons as more clients join. That makes MPC's latency profile brutal at any FL scale beyond small consortiums.

Here's what connects all three approaches, and it's easy to miss because each technique gets judged on its own terms: none of them protects against a compromised or malicious hypervisor, or a cloud administrator with root access to the machine actually running the computation. Data in use, meaning data sitting decrypted in memory while a CPU or GPU works on it, stays exposed no matter how well it was encrypted in transit or at rest. Differential privacy in particular cannot offer full confidentiality against a malicious hypervisor, and a malicious hypervisor is exactly the threat model that applies to any FL deployment running on public cloud infrastructure the organization doesn't fully control.

So the gap isn't really in the cryptographic protocols. It sits one layer down, at the hardware execution layer. What's missing is something that enforces confidentiality while the computation is actually happening, not just before the data gets sent or after the result comes back.

What a TEE actually guarantees and where CPU-only TEEs fall short for ML

A trusted execution environment gives three specific guarantees, and it's worth pulling them apart because vague talk of "secure enclaves" tends to blur them together. Confidentiality means the program's execution and its in-memory state stay invisible to everything outside the enclave, including the host operating system, the hypervisor, and whoever administers the cloud account. Integrity means the code running inside can't be tampered with or silently redirected mid-execution. Remote attestation means a party outside the enclave can cryptographically check exactly which code is running, and in what starting state, before handing over anything sensitive, whether that's a dataset or a set of model weights.

The CPU TEE landscape splits by scope. Intel SGX protects individual processes at the enclave level. AMD SEV-SNP and Intel TDX protect entire virtual machines, which is the more practical unit for cloud workloads since it doesn't force a rewrite of the application to fit inside a narrow enclave interface. ARM TrustZone targets embedded and mobile devices, which matters a lot for FL client-side deployment, since plenty of FL clients are phones, not servers.

One system worth a close look here is GradSec, presented at Middleware in 2022 in Quebec, Canada. Rather than stuffing an entire ML model inside a TrustZone enclave the way earlier approaches like DarkneTZ did, statically loading a large chunk of the model into the trusted zone, GradSec picks out and protects only the most sensitive layers. That selectivity cut the trusted computing base by up to 30% and cut overall training time by up to 56% against the older, blunter approach. It's a clean illustration of a broader rule in TEE design: protecting everything indiscriminately costs a lot, and the performance gains hide in figuring out what actually needs protecting.

But here's the ceiling every one of these CPU TEEs runs into, and it's the one that matters most for the rest of this piece. SGX, SEV-SNP, and TDX were built for general-purpose CPU compute. They protect what happens on the CPU. The actual tensor math for any model worth training, the matrix multiplications that dominate FL workloads, runs on a GPU. The moment data crosses the PCIe bus from CPU to GPU, it steps clean outside the enclave's protection boundary.

Picture an FL aggregator running inside a properly configured AMD SEV-SNP confidential VM. It looks protected: attestation, encrypted memory, all the right hardware features flipped on. And it's still sending plaintext gradient data across the CPU-GPU interface, where an untrusted hypervisor sitting between the two can intercept it. That's not a hypothetical gap. That's the exact gap GPU TEEs exist to close.

How GPU TEEs extend hardware confidentiality to the compute itself

NVIDIA's H100, part of the Hopper generation released in 2022, was the first production GPU to ship with a confidential computing (CC) mode and an on-die root of trust. CC mode support went generally available with CUDA 12.4. What it does, concretely, is protect GPU memory, the GPU's configuration registers, and every byte of CPU-GPU communication, encrypting and integrity-checking data as it crosses the PCIe link in both directions.

The H100's confidential computing mode includes verification mechanisms at initialization, closing off a whole class of attack where malicious code gets slipped in before the trusted environment is even set up. Hardware attestation goes a step further: the GPU produces a cryptographic report, cryptographically bound to the hardware, supporting verification that the GPU is running the software stack it claims to be running. A participant in an FL round can check that report before agreeing to send a model or a gradient anywhere near that GPU.

The H200, released in 2024, sits in the same Hopper family and improves on memory bandwidth and CC-mode performance without changing the underlying architecture story. On the AMD side, one accelerator, released in 2024, introduced Infinity Guard for chip-level isolation, its own answer to the same problem.

Intel's contribution here is TDX Connect, first disclosed in 2023 with hardware support announced for Xeon 6 in 2025, developed alongside NVIDIA. TDX Connect handles secure communication between a confidential VM on the CPU side and a confidential-computing-capable GPU, stretching the trust boundary across the interface instead of leaving a gap there. NVIDIA has also carried CC mode support forward into its Blackwell generation, worth noting because it turns confidential computing into a standing platform feature across GPU generations, not a one-off experiment tied to Hopper alone.

What actually changes here is where trust gets anchored. It used to stop at the CPU package boundary. Now the TEE spans the CPU confidential VM and the GPU as one continuous unit, which locks the cloud provider out of the whole computation, not just the CPU-side half of it.

How confidential GPU TEEs restructure the federated learning architecture

Two points in the FL pipeline are where GPU TEEs actually change what's possible. The first is the aggregation server. Run the aggregator inside a Hopper CC-mode enclave, and gradient updates are processed inside a hardware-protected boundary. The cloud provider hosting that server is locked out of individual client updates and the aggregated result alike. No single party, not even whoever operates the infrastructure, ever holds the plaintext model parameters.

The second is client-side training. Each participant runs local training inside a confidential GPU VM, which does two things at once: it protects the integrity of that local computation, and, through attestation, it lets the server verify a given client ran the code it was supposed to be running.

That's a real shift in the trust model, worth sitting with for a second. Participants no longer need to trust whoever operates the aggregator, as an organization or as a person. They trust the TEE's attestable code, which is a far more verifiable thing to trust than a company's word. That's what makes it possible to run FL aggregation on public cloud infrastructure without handing that cloud provider access to gradients or model weights, something the earlier architecture made structurally impossible.

Byzantine resistance gets a boost from the same mechanism, through a different route. Because each client's training environment is attested, the server can reject any update that didn't come from a verified, unmodified training process. That's a system-security approach to filtering bad actors, built on cryptographic proof of what code ran, rather than a statistical approach that eyeballs updates for how they look. Sentinel, covered further down, builds directly on this idea.

There's a regulatory angle here too, and it shouldn't get buried under the technical detail. Data protection regulation restricts personal data from leaving a designated economic region without specific legal cover, such as adequacy decisions or standard contractual clauses. China's PIPL imposes a comparable restriction on data leaving China. Confidential FL lets organizations across these jurisdictions train a shared model together without any raw data, and without even plaintext gradients, ever leaving each organization's own controlled environment. That's not a side benefit. For a multinational healthcare or financial consortium, it can be the difference between a project being legal and not.

NVIDIA FLARE implements this with three mechanisms working together: explicit trust checks through attestation at defined stages of each FL round, secure aggregation where the FL server itself runs inside a TEE so individual client updates never surface outside the enclave, and model IP protection, where TEEs on every node stop anyone from extracting or reverse-engineering model weights at runtime.

Performance overhead: what the empirical evidence actually shows

None of this matters if the overhead makes it unusable. So what does the overhead actually look like, measured against real workloads?

For single-GPU inference on an H100, the picture looks good. Research from Phala Network (Zhu, Yin, Zhou, 2024) found that for most typical LLM queries, CC mode overhead stays under 5%. The bottleneck turns out to be CPU-GPU data transfer rather than the encryption overhead itself. The bottleneck turns out to be CPU-GPU data transfer across PCIe, not the GPU's own internal computation, which lines up with the earlier point about where the vulnerable gap actually sits.

On the CPU TEE side, a 2025 paper (Chrapek et al., arXiv 2509.18886) ran Llama 2 in its 7B, 13B, and 70B parameter sizes inside Intel TDX and SGX, accelerated with Advanced Matrix Extensions (AMX). Throughput overhead stayed under 10%, latency overhead of around 20%, and AMX support pushed both figures down further. Inference workloads, in other words, are close to a solved problem at this point.

Multi-GPU distributed training is where the real difficulty lives. Every piece of data leaving a GPU package under TEE has to get AES-GCM encrypted and MAC-authenticated by the sender, then decrypted and verified by the receiver on the other end. That encryption tax touches every phase of ring-all-reduce, the communication pattern most distributed training relies on to sync gradients across GPUs.

A 2025 study (Lee et al., arXiv 2501.11771) measured this directly across four GPU TEEs and found total runtime jumped several times over for GPT-Xlarge, with the average across tested models landing in the same range, compared to standard distributed data parallel (DDP) training without TEEs. The overhead gets worse as GPU count and model size climb, since bigger models trigger more asynchronous all-reduce operations, and each one carries its own encryption tax.

There's a mitigation worth naming specifically because it's an available lever right now, not a research promise. Careful tuning of communication batching parameters can help reduce TEE-induced inter-GPU overhead by reducing the number of separate encrypted exchanges. The honest summary: inference under CC mode is ready for production at single-GPU scale today. Confidential multi-GPU training still carries real, measurable overhead that needs active tuning, and it stays an open research question rather than a solved engineering problem.

Three research systems that show where the field is heading

EnclaveX, a system out of TU Dresden with Scontain and STACKIT (arXiv 2606.31408, dated 30 June 2026), combines Intel TDX at the VM level with NVIDIA H200 GPUs, plus SCONE for protection at the application layer. By the authors' account, it's the first system to stretch process-based TEE isolation, the SGX style of protection, over confidential GPUs running inside confidential VMs.

The gap EnclaveX closes is a subtle one that VM-level confidential computing leaves open by default. An administrator with elevated platform privileges can exec directly into a confidential VM and get at the data inside it, even when the VM's hardware protections are configured correctly, because that access sits above the hardware boundary entirely. EnclaveX handles this by only releasing encryption keys after successful attestation at the application layer, and by disabling memory dump features inside the guest kernel so an administrator can't just snapshot memory and walk off with it. The overhead for all this attestation machinery comes out to roughly 1.1758 seconds total, about 1.27%, negligible for nearly any real workload.

GPU Travelling, presented at ACM CCS 2025 in Taipei (October 13 to 17, published in the proceedings at pages 2653 to 2667), takes an odd angle on the communication problem. Instead of moving a dataset to where the GPU lives, the confidential GPU itself travels to the data holder, loads the dataset directly into protected GPU memory on-site, trains there, then returns. That cuts out data transmission for training entirely, and it matters more every year as LLM training datasets grow large enough that shipping data and model back and forth gets prohibitively expensive on bandwidth alone. The confidentiality guarantee here stretches to the level of the data center itself, not just the individual GPU.

Sentinel, out of Virginia Tech (arXiv 2509.00634, August 2025), tackles the Byzantine problem from the system-security side instead of the statistical side mentioned earlier. Code instrumentation tracks control flow and critical variables all through local training, and a trusted training recorder living inside the TEE produces a cryptographically signed attestation report that rides along with the gradient update to the aggregator. The server checks that report before accepting anything. Updates from training environments that can't produce a valid, unmodified attestation get rejected before they ever reach the aggregation step, sidestepping the false-positive problem that dogs defenses trying to spot bad actors purely by the statistical shape of their updates.

What end-to-end confidential FL requires beyond the GPU hardware

Hardware closes one gap, and it's a big one. But it's worth being honest about what it leaves open. H100 and H200 CC mode solves the CPU-GPU interface problem. It does not automatically protect the orchestration layer sitting above it, the model serving pipeline, or the toolchain used to deploy any of this in the first place.

EnclaveX makes this concrete: a Kubernetes administrator with API access can still breach a confidential VM even when every piece of underlying hardware is configured exactly right. The hardware did its job. The orchestration layer above it was never in scope for that protection in the first place. Application-layer attestation and key management have to sit above the VM watching that layer too, or the hardware guarantee ends up decorative.

Attestation itself needs to run continuously and in layers, not as a one-time check at startup. A participant contributing to an FL round needs proof not just that a GPU is genuine, but that the aggregation code running on it is the correct version, that the model being trained is the correct model, and that every client's training code matches what was agreed on, all before any update gets accepted. That's a chain running from the CPU confidential VM, through the GPU, up to the application, and every link in it has to hold.

On the client side, not every device in an FL deployment runs H100-class hardware. Phones and IoT devices show up constantly as FL participants, and for those, the ARM TrustZone approach GradSec demonstrates offers a workable answer: protect the most sensitive layers of the model selectively, instead of trying to cram the whole thing into a resource-constrained enclave, keeping the trusted computing base small and training time manageable.

If there's one highest-leverage move in all of this, it's putting the FL aggregator inside a TEE, full stop. That single intervention does two things at once: it keeps individual client gradients hidden from whoever operates the aggregator, and through attestation, it gives every client a way to check exactly what code is processing their contribution before they send it anywhere. That's the point where privacy-preserving FL stops being a line in a whitepaper and becomes something a participant can actually verify. For any organization building federated learning on public cloud infrastructure it doesn't fully control, that's the practical lesson: confidentiality has to get built into the compute stack from the start. Bolt it on after the architecture is already locked in, and the same gaps described at the start of this piece show up again, just one layer further down.