Cold Start Attestation Latency in Serverless Confidential Inference

Cold start latency in serverless confidential inference isn't one delay. It's six of them, stacked in a fixed order where each layer has to finish before the next one can even begin. Knowing that sequence, and knowing which layer actually eats the time, separates "confidential inference is just slow" from finding the one place worth spending engineering effort on. Most teams get this wrong: they optimize model loading because it's the biggest, most visible number, while the real damage happens two layers earlier, in a chain they never touch.

Serverless economics run on scale-to-zero: idle instances cost nothing, but the first request after a quiet stretch pays for everything that scaling to zero put off. Research has found cold start latency can eat the majority of total response time in FaaS workloads, with no confidential computing involved at all, so the baseline problem was already ugly before TEEs showed up. Add a trusted execution environment, and two new layers get wedged into a sequence already dominated by one expensive step: loading model weights. What comes out the other end isn't a bit slower. It's a structurally different cold start, one where privacy-preserving inference and scale-to-zero economics start pulling against each other. That fight is worth taking apart layer by layer.

What the six sequential layers of a confidential cold start actually are

Layer one is GPU allocation and container scheduling. Nothing TEE-specific happens here. It's the same hardware provisioning any serverless GPU workload deals with, but it's the layer that starts the clock.

Layer two is confidential VM boot, and this is where things split off from a normal cold start. Every page of guest memory has to be cryptographically checked before the VM can run, a step called pvalidation. Non-confidential serverless has nothing like it.

Layer three is the remote attestation handshake. An Attester collects hardware evidence, ships it to a Verifier, the Verifier checks it against policy, and a signed result comes back. This runs 1 to 3 seconds for the SDK call plus OCSP verification during provisioning. It happens once per cold start, not once per request, so it disappears from steady-state throughput math entirely. But at startup it can't be skipped, and everything downstream waits on it.

Layer four is attestation-gated key release. A KMS call releases the model's decryption keys, and it only fires if attestation succeeded. No attestation result, no keys. No keys, no model. This is the layer that turns three separate steps into a dependency chain, and it's the one people underestimate most, because on its own it looks like a quick API call rather than a gate.

Layer five is model weight loading into encrypted VRAM, the layer that dominates cost for large models and grows with model size. Layer six is warm-up: CUDA graph capture, kernel JIT compilation, KV-cache initialization, still sequential, still standing between the system and the first real inference.

Layers two through four are where the trouble concentrates, because they're strictly sequential and each one gates the next. There's not much room to parallelize anything while attestation sits in the critical path. The trust boundary doesn't stop at the CPU either; it runs across the PCIe bus into the GPU itself. An NVIDIA H100 or H200 running in confidential computing mode needs a compatible host CPU TEE underneath it, either AMD SEV-SNP or Intel TDX, and neither one alone covers the whole inference pipeline. CVM boot and GPU attestation are coupled steps, not two things happening side by side on their own.

How each TEE hardware stack distributes cost across those layers differently

Not all TEE stacks spend their overhead in the same place, and that matters more than most hardware comparisons let on.

On AMD SEV-SNP, the cost lives almost entirely in CVM boot. SEVeriFast (Holmes, Waterman, and Williams, ASPLOS 2024) found Linux boot takes significantly longer under SEV than without it, with pvalidation of guest memory pages as the dominant cost inside that number. Turning on huge pages dramatically reduces average pvalidation time, which tells you the entire boot penalty was hiding in one fixable mechanism the whole time.

Intel TDX looks different on its own. A 2025 study running full Llama 2 pipelines (7B, 13B, and 70B parameters) inside TDX and SGX measured VM overhead of 1.82% to 5.38% against a VM baseline using 2MB transparent huge pages. Field-reported numbers from H200-plus-TDX deployments put measured latency overhead around 5.2%. Modest, on its own, but that phrase is doing a lot of work.

Stack TDX and GPU confidential computing mode together, and the picture changes for the worse. EnclaveX (June 2026) found TPS overhead running 35.0% to 62.8%, with Time Between Tokens overhead between 35.2% and 73.9%, against native non-confidential inference. That's a wide margin worse than CPU-only TDX, because it compounds two costs at once: TDX's CPU virtualization tax plus GPU CC mode's own tax. The overhead does shrink as batch size grows, which matters for anyone planning capacity around it, but it's a real warning against assuming CPU-side numbers tell you anything about the combined system.

NVIDIA's H100 GPU CC mode by itself looks considerably lighter. A 2024 study found average steady-state overhead under 9% across tested models. On Llama-3.1-8B specifically, TEE-on throughput measured 123.3 tokens per second against 132.4 with TEE off, a 6.85% hit in tokens-per-second and 3.22% in queries-per-second. That gap narrows further as models scale up: Llama-3.1-70B showed close to zero overhead on H100. Under concurrency, first-token latency ran 71 milliseconds with TEE versus 65 without at a concurrency of 1 (a 9.23% gap), widening to 16.18% at 64 concurrent requests, though overall throughput overhead under high concurrency remained relatively contained.

VRAM encryption itself appears to add comparatively little overhead on top of the raw cost of moving weight data around, based on available benchmark evidence for GPU confidential computing mode.

Line those numbers up and the lesson is blunt: picking the newest GPU generation matters less than picking the right split between CPU TEE cost and GPU CC mode cost. Chasing chip generation over that split is a mistake, and it's a common one. H100-only confidential computing paired with a lighter host setup is a completely different cost profile than a full TDX-plus-H200 stack, and teams that buy the newer GPU without checking which CPU TEE it's sitting on are optimizing the wrong variable.

Where attestation latency sits in the sequence and why its position matters more than its duration

One to three seconds sounds trivial next to the minutes some model loads take. That instinct is wrong, and it's wrong because of position, not duration. Attestation gates key release, key release gates model loading, so a delay at layer three doesn't just cost 1 to 3 seconds on its own. It pushes back the start time of every layer that comes after it.

For warm instances already running, this cost gets paid once and stays invisible, since attestation fires at cold start and steady-state requests never touch it again. But true scale-to-zero deployments don't get to spread that cost out, because every cold start re-triggers the full attestation flow from scratch. And bursty, unpredictable traffic, exactly the pattern serverless is supposed to handle well, is exactly the pattern that hits cold starts often instead of rarely.

Where does the 1 to 3 seconds actually go? The Attester collects evidence locally, which is fast. Quote generation depends on the hardware. Then there's a network round trip to a Verifier service, a policy check on the Verifier's end, and the result comes back. Teams building these systems have to pick a verification endpoint: AMD's Key Distribution Service for VCEK certificates, NVIDIA's Remote Attestation Service for GPU quotes, or a third-party broker like Intel Trust Authority. That choice shapes both network latency and how much the whole system leans on an external service staying up.

GPU deployments add one more wrinkle. On NVIDIA H100 and H200 hardware, the CVM and the GPU set up a secure channel through the SPDM protocol, and the H100's hardware root of trust and device attestation mechanism have to get verified as part of that handshake. That's an extra sub-step buried inside layer three with no equivalent in CPU-only TEE setups.

One might argue the fix is obvious: just make attestation faster. That's the wrong target, and chasing it is how most of this research ends up pointed at the wrong layer. Most serverless LLM work optimizes model loading and leaves CVM boot, attestation, and key release almost entirely untouched, so a faster model loader sitting behind a slow, gated attestation chain still just waits its turn. The real waste isn't in the 1 to 3 seconds. It's in every downstream layer sitting idle because nobody questioned why attestation has to happen first at all, a question the last section of this piece takes seriously.

What research has achieved in compressing the CVM boot and attestation layers

SEVeriFast, again from Holmes, Waterman, and Williams at ASPLOS 2024, goes straight after the CVM boot layer, and what it found runs against intuition. Rather than stripping bootstrap stages down further, the researchers added a bootstrap component back in and reintroduced kernel compression, and that combination optimized cold boot performance for SEV VMs better than the leaner approach did. The payoff: average boot time dropped 93.8% for the Lupine kernel, 88.5% for the AWS kernel, and 86.1% for the Ubuntu kernel. Paired with the huge-pages fix mentioned earlier, pvalidation time falling from over 60 milliseconds to under 1 millisecond, the paper makes a strong case that CVM boot, something that looks like a fixed cost baked into the silicon, is actually quite engineerable.

A separate line of work, WorksetEnclave (Yan et al., ASPLOS 2026), goes after SGX serverless cold starts through a different mechanism entirely: workset-tracked snapshots. Instead of re-initializing an entire enclave from scratch, the system tracks which memory pages actually get touched during execution and checkpoints only those. Restoring an enclave means replaying just that working set instead of rebuilding the whole thing. Results: cold start acceleration between 1.9x and 54x, enclave memory use cut by 13.37% to 94.87%. The approach shipped in two prototypes, one on Gramine LibOS and one on Occlum, so it isn't glued to a single runtime. The paper is upfront about what's still broken, though: cold starts still happen often whenever concurrent requests outnumber deployed instances, and memory overhead under high concurrency stays a real problem by the authors' own account.

These two projects don't overlap at all. One targets SEV, the other SGX. One compresses boot, the other compresses enclave restore. Nothing published so far covers the whole upper half of the cold start sequence in one pass, which means anyone serious about shrinking confidential serverless cold starts needs to combine techniques from both categories rather than picking a favorite.

What research has achieved in compressing the model loading layer, and what changes when VRAM is encrypted

Model loading is still the single biggest line item in a large LLM's cold start. It grows close to linearly with parameter count, and it stays the bottleneck even after every other layer gets tuned to death.

ServerlessLLM tackles this with a multi-tier checkpoint loading system that spreads weight storage across GPU memory, DRAM, and SSDs, loading in parallel chunks instead of one after another. That design hits up to 8.2x faster initialization against baseline loading, and the same system adds live inference migration, moving only the minimal state needed between servers, plus a scheduler tuned specifically for startup time. ParaServe takes a different angle on the same layer, overlapping parameter loading across multiple GPU servers and using pipeline parallelism to cut fetching and initialization delay, rather than leaning on storage tiering. Other approaches come at it from different directions, targeting initialization overhead through various profiling and templating strategies.

GPU memory snapshots might be the most structurally interesting idea on the table. The concept: capture the entire GPU state as a byte-for-byte snapshot, weights sitting in VRAM, CUDA kernel objects, execution context, all of it, in one shot. Restoring from that snapshot means the GPU just replays the saved state, skipping weight loading, CUDA graph capture, and context setup entirely. CRIU supports this at the Linux process level with recent NVIDIA drivers, and Modal shipped an alpha version of GPU memory snapshots in 2025.

But how does this hold up against encrypted VRAM? That's genuinely unresolved in the published work so far, and it's worth sitting with rather than glossing over. A snapshot taken from a TEE holds encrypted bytes tied to that specific instance's keys, and a freshly cold-started instance comes up under new attestation with new keys. Restoring an old snapshot into a new key context is a real key management problem, not a minor performance detail, and nothing in the current research says how that gets solved. This is arguably the single biggest open question in the whole stack: every gain from ServerlessLLM, ParaServe, or Modal's snapshot approach assumes plaintext VRAM, and none of it has been proven to survive the encryption layer this entire piece is built around. Anyone pitching a snapshot-based fix to a confidential inference cold start without addressing that gap is selling a solution to a different problem than the one on the table.

One more piece worth mentioning, even a layer up from raw weight loading: FaaSLight, from researchers at Peking University, targets code loading rather than model loading, pulling in only the code a function actually needs at startup and pushing off the rest. It reports up to a 78.95% cut in code loading latency (28.78% on average), a cut in total response latency of up to 42.05% (19.21% on average), and a 21.25x improvement over the prior state of the art. None of it was built with attestation-gated key release in mind, so wiring any of these model-loading tricks into layer four's dependency on a successful attestation result is still an open systems design question, not just a tuning exercise.

An emerging alternative attestation model for on-device and edge inference that rethinks where attestation happens

Everything covered so far treats attestation as something done to the boot sequence: verify the hardware, then release keys, then load the model. Remote attestation as a technique was built for VMs and containers, and it strains when the thing being attested is a billion-parameter neural network instead of a virtual machine image. Time and memory efficiency both take the hit.

AttestLLM, published September 2025 and updated February 2026, proposes something structurally different. Instead of attesting the boot environment and then loading a model, it embeds a watermark signature directly into the activation distributions of an LLM's internal building blocks, so the model itself carries proof of its own authorization. Attestation stops being a step in the boot sequence and becomes a property of the model. The framework exists to protect hardware vendors' IP, making sure only authorized models run on a given device, and the deployment splits work across a Rich Execution Environment handling inference, a minimal privileged hypervisor at EL2 (pKVM) enforcing memory isolation through Stage-2 MMU, and a TEE enclave with a footprint of at least 512 MB doing the actual attestation work, small enough to fit inside the tight memory limits of Arm TrustZone.

Tested against Llama, Qwen, and Phi models up to tens of billions of parameters under INT4 and INT8 quantization, it hit near-perfect watermark extraction accuracy on authorized models, with average perplexity degradation that remained minimal and zero-shot accuracy degradation that was negligible against non-watermarked versions.

What that means for the cold start sequence traced through this piece deserves a beat of thought. If attestation lives inside the model itself rather than gating a separate boot-and-key-release chain, the strict sequential dependency running through layers two, three, and four starts to come loose. That doesn't make the cold start problem disappear for confidential serverless inference generally, and this approach targets on-device and edge deployment rather than the datacenter TEE stacks covered earlier, so it isn't a drop-in replacement. But it does say attestation doesn't have to sit where current architecture puts it, and that's the real lesson buried under all the benchmarks in this piece. Most of the research here works to make the gate faster. Almost none of it asks whether the gate belongs in the critical path at all. Sometimes the fastest way to shrink a bottleneck isn't speeding up the step. It's asking whether the step needs to happen where everyone assumed it did.