Confidential Containers with Kata Containers and TEE Backends

The phrase "trusted execution environment" gets used loosely enough that its meaning has blurred in much of the industry writing around it. At the hardware level, a TEE is a CPU-enforced memory boundary. The memory inside it is encrypted and integrity-protected by the processor itself, not by any software stack running on top. Encryption keys are held within the CPU and never surfaced to the host operating system, the hypervisor, or any software component outside the protected region. That is not a policy statement; it is a physical constraint of the memory bus. But what if the software stack itself is compromised before the TEE boundary is even established — does the hardware guarantee still hold?

Two structurally different isolation models exist in the current TEE landscape. Process-based isolation, of which Intel SGX is the canonical example, splits a single application process into trusted and untrusted components. The trusted component executes inside an encrypted memory enclave; the untrusted component interacts with the broader operating system on its behalf. The boundary is fine-grained and the trusted computing base can be made extremely small. The cost is developer burden: an application must be explicitly partitioned to run in this model, a non-trivial engineering investment for most cloud-native workloads. I have watched teams underestimate that porting cost badly, spending months on enclave decomposition before concluding the operational lift was not justified by their threat model.

VM-based isolation takes a different approach. Intel TDX (Trust Domain Extensions), AMD SEV-SNP (Secure Encrypted Virtualization with Secure Nested Paging), and IBM Secure Execution each protect an entire virtual machine. The guest OS, its kernel, and every process running within it share encrypted, integrity-protected memory. The TEE boundary is the VM boundary, and workloads run without modification inside the guest. The application does not need to know it is operating within a TEE at all. For containerized workloads that were never written with enclave partitioning in mind, this is a far more practical entry point.

Remote attestation is inseparable from any of this. A TEE that cannot be attested is effectively just encrypted memory: no way to verify what is running inside it, no basis for trust. The CPU generates cryptographic evidence, a signed measurement of exactly what code, firmware, and configuration is present inside the TEE at a given moment, that a remote party can verify before releasing any secret into the enclave. Without that verification step, the encryption provides confidentiality but not verifiable integrity. A workload owner cannot distinguish a genuine, unmodified enclave from one running compromised software, which makes the guarantee hollow from the perspective of someone deciding whether to trust the environment with a private key or a sensitive dataset.

How Kata Containers creates the VM isolation layer that TEEs build on

Kata Containers replaces the conventional container runtime with a model in which each pod executes inside a lightweight virtual machine, managed by a hypervisor, typically QEMU or Cloud Hypervisor. From Kubernetes' perspective, a Kata pod is indistinguishable from an ordinary pod. The scheduler, the orchestration layer, and the CNI plugin interact with it identically.

In the standard Kata model, without any confidential computing extension, container images are pulled on the worker node and the resulting filesystem bundles are passed into the guest via virtiofs or 9p. This provides meaningful isolation between workloads at the VM boundary, stronger than the namespace and cgroup isolation a conventional container runtime offers, but it does not provide confidentiality from the host. The container image content, and by extension the workload's filesystem, remains visible to the host OS.

The Kata Agent is a daemon that runs inside the guest VM, receiving instructions from the Kata Shim process on the host over a channel that crosses the VM boundary. This host-to-guest API is the central nervous system of the architecture; nearly everything the host communicates to the workload passes through it.

What Kata provides, and what makes it the right foundation for confidential computing, is a well-defined VM boundary that hardware can actually seal. A process running in a container namespace has no such boundary, nothing for the CPU's memory encryption engine to anchor to. The VM is the meaningful unit that TDX, SEV-SNP, and IBM Secure Execution can protect. Without Kata's VM layer, there is no surface for the TEE to act upon.

Confidential Containers does not replace Kata; it extends it. Kata provides the VM. CoCo instructs Kata to boot that VM inside a TEE, moves image pulling from the host into the guest, and layers attestation machinery over the entire assembly. The developer-facing change is minimal: selecting the appropriate runtimeClassName in a pod specification is the primary operational difference from a conventional Kubernetes deployment. The architectural implications beneath that single field are substantial.

What changes when Kata boots inside a TEE: the CoCo guest environment

Once Kata boots the guest VM inside a TEE, the properties of that execution environment change in ways that matter concretely to a security architect. The host, the hypervisor, and the cloud operator are excluded from reading or modifying the guest's memory. A fully compromised host OS cannot extract workload data because the data does not exist in accessible form outside the encrypted memory region.

Image handling is the first architectural change that surfaces in practice. Standard Kata pulls container images on the worker node; CoCo cannot allow that, because pulling on the host exposes image content to an environment that is, by design, untrusted. Instead, images are pulled and unpacked inside the guest VM by a component called image-rs, and the CRI runtime on the host is deliberately configured to skip image pulling entirely. This is not a performance tuning choice; it is a necessary structural change to maintain the confidentiality guarantee.

The minimal guest operating system that boots inside the TEE is itself a trust anchor. Its firmware, rootfs, kernel, and kernel command line are all measured, meaning the CPU computes a cryptographic digest of each component as it loads. Any tampering alters the measurements. A manipulated guest produces different measurement values and therefore fails attestation, which is the mechanism by which the integrity of the execution environment is verified before any secret is released into it.

Two components inside the guest carry out the operational work of attestation and secret retrieval. The Attestation Agent initiates and manages the attestation flow on behalf of the guest, communicating with external verification services. The Confidential Data Hub brokers access to secrets once attestation has succeeded; a workload retrieves keys and sensitive configuration through this component, not through any direct channel to an external store. The separation of concerns is intentional: attestation establishes trustworthiness, and only then does secret provisioning begin.

The net effect on the trusted computing base is significant. The workload no longer depends on the integrity of the host OS, the kubelet, containerd, or any other host-side component. Trust is reduced to the TEE hardware itself and the measured software stack running inside the guest, a substantially smaller, more auditable surface than what a conventional Kubernetes node presents.

The TEE backend options and what each one implies for the deployment

The CoCo operator exposes distinct runtime classes for each backend combination, making the TEE selection explicit at the pod level: kata-qemu-tdx, kata-qemu-snp, kata-qemu-sev, kata-clh-tdx, and others. Backend choice is not a purely technical decision; it is substantially constrained by what hardware the deployment environment actually provides, and in practice that constraint arrives before any architectural preference does.

Intel TDX operates at the VM level. The entire trust domain, corresponding to the guest VM, runs in encrypted, integrity-protected memory. It pairs with either QEMU or Cloud Hypervisor in the Kata stack, and attestation evidence takes the form of a TD Quote, a hardware-signed structure encoding the measurements of the trust domain that is verified by an attestation service.

AMD SEV-SNP adds integrity protection on top of the memory encryption that earlier SEV variants provided. The Secure Nested Paging component matters because earlier SEV and SEV-ES variants protected confidentiality but were vulnerable to certain memory remapping attacks; SNP closes those gaps. SEV-SNP has seen strong adoption in cloud provider bare-metal offerings, making it one of the more practically accessible backends for teams operating in public cloud environments, and the kata-qemu-snp runtime class is the relevant entry point.

IBM Secure Execution targets IBM Z (s390x) platforms, and the Protected Execution Facility addresses IBM POWER architectures. These are less common in commodity cloud deployments but significant for enterprise workloads already operating in mainframe environments, where regulatory and operational requirements for strong isolation are often the most stringent.

Intel SGX occupies a different position in this landscape. It operates without a utility VM; the trusted component runs as an encrypted enclave at the process level. The Attestation Service in the CoCo stack can validate SGX evidence, but the operational model is substantively different from the VM-based backends. There is no Kata VM involved. SGX is relevant for scenarios requiring fine-grained enclave control at the process level rather than whole-pod isolation, and it carries the developer porting burden described in the first section.

ARM CCA and Hygon CSV are also supported by the Attestation Service. It is also worth considering what their presence in the stack signals: an intent to remain hardware-agnostic as the TEE ecosystem evolves, rather than committing the architecture to any single vendor's implementation, which matters more than it might appear given how quickly the hardware landscape has shifted over the past several years.

How remote attestation works in CoCo: the Trustee stack and the RATS model

The IETF Remote ATtestation procedureS (RATS) architecture defines a framework for how an attesting environment proves its trustworthiness to a relying party. CoCo follows the RATS "background check" model: the confidential guest VM is the Attester, the Key Broker Service is the Relying Party, and a dedicated Attestation Service performs the actual evidence verification. Each role is well-defined and separable, which matters in operational deployments where the entity controlling secret release may be distinct from the entity managing reference values. In multi-party data processing scenarios, that separation is not an academic nicety; it is often what makes the arrangement contractually viable at all.

Trustee is the CoCo project's reference implementation of this stack, composed of three cooperating services. The Key Broker Service (KBS) is the HTTP endpoint that confidential guests contact; it orchestrates the attestation exchange and conditionally releases secrets when policy permits, but deliberately delegates evidence verification rather than performing it internally. The Attestation Service (AS) accepts raw hardware evidence, TEE quotes and measurements, validates them against reference values, and returns a structured attestation result. The Reference Value Provider Service (RVPS) supplies the expected measurements: known-good firmware hashes, kernel digests, and other reference values that the AS compares against what the TEE actually reports. Registering reference values in the RVPS is how a platform operator encodes what a trustworthy guest looks like, and keeping those reference values current as guest components are updated is an ongoing operational discipline, not a one-time configuration exercise.

The image decryption flow illustrates how the components interlock in practice. VM attestation runs first, confirming the integrity of the underlying execution environment, including firmware, kernel, and guest rootfs. Container attestation follows, confirming the container runtime components inside the guest. Only after both verification steps succeed does the runtime request decryption keys from the KBS for encrypted container images. Those keys never transit through the host; they are provisioned directly into the trusted guest environment through the Confidential Data Hub.

Attestation result tokens follow the EAT Attestation Result (EAR) message format, a structured and signed representation of claims about the guest's trustworthiness. The use of a defined message format matters for interoperability: different Attestation Service implementations can produce compatible results if they share the format, which reduces lock-in to any particular vendor's verification stack.

Trustee manages two distinct policy layers, and the separation is operationally significant. KBS resource policies control which secrets are released and to which workloads; Attestation Service policies define how measured TCB claims are evaluated against reference values to determine enclave validity. A workload owner can control secret release independently of the platform operator's reference value definitions. In a multi-party scenario, this allows the party that owns the sensitive data to set their own release conditions without depending on the infrastructure operator's attestation policies, which is the kind of boundary that legal and compliance teams often require before they will approve the arrangement.

How the Kata Agent policy secures the host-to-guest API boundary

Attestation establishes trust at the moment of VM boot. It does not govern what happens afterward. The Kata Agent inside the guest continues to receive API calls from the Kata Shim on the host throughout the pod's lifetime, and that host is, by architectural design, untrusted. A malicious or compromised host could attempt to manipulate a running workload by sending crafted API requests across the host-to-guest channel: injecting an unexpected container, modifying environment variables, altering mount configurations. Attestation of the initial state provides no protection against this class of attack, and in my experience this is the gap that gets underestimated most often when teams first reason through the threat model. That raises an important question: if attestation only verifies the state at boot, what prevents a hostile host from quietly manipulating the guest through legitimate-looking API calls afterward?

CoCo's answer is an Open Policy Agent (OPA) instance running inside the guest, evaluating every incoming API request against a declarative rule set before the Kata Agent acts on it. The policy is expressed in Rego, OPA's policy language. Rules are scoped to the specific pod: expected container images, environment variables, mount points, capabilities, and other runtime parameters are all explicitly encoded. Any request that deviates from the declared policy is blocked inside the guest. The host cannot override this enforcement because it runs within the TEE's protected memory.

The policy document is bound to the TEE at VM creation time through a mechanism that matters cryptographically. The Kata Shim computes a SHA-256 hash of the policy and attaches it to the TEE measurement before the VM starts. That hash becomes part of the attestation evidence, which means it is cryptographically committed. A workload owner verifying attestation can confirm not only the guest OS and firmware measurements but also the exact policy governing the host-to-guest API for that specific pod. Replacing the policy after the fact produces a different measurement and fails attestation.

The genpolicy tool automates policy generation from an existing Kubernetes YAML manifest, which meaningfully reduces the operational burden of producing correct per-pod policies. Writing Rego by hand for each deployment is error-prone; deriving it from a known-good pod specification is considerably more tractable, though it still requires someone to review the output and understand what they are committing to.

The combination of OPA enforcement inside the TEE and the policy hash committed to the attestation measurement closes what would otherwise be a persistent gap. Attestation without API boundary enforcement leaves the guest exposed to post-boot manipulation. API boundary enforcement without cryptographic commitment to the policy allows a host to substitute a permissive policy for a restrictive one. The two controls are mutually dependent; the architecture requires both.

Peer Pods: reaching confidential computing on cloud VMs without bare metal

Most TEE hardware does not support nested virtualization. A confidential guest cannot be started inside an already-virtualized environment, which is precisely the situation of a conventional cloud Kubernetes node: the node itself runs as a VM on the cloud provider's infrastructure. This constraint would ordinarily confine CoCo to bare-metal worker nodes, a significant limitation for teams whose entire infrastructure lives in a public cloud and who have no path to bare metal within their current contracts or operational model. I have seen this hardware constraint stop CoCo evaluations before they properly started.

The Peer Pods model resolves this through a structural change rather than a hardware workaround. Instead of launching the pod sandbox as a nested VM on the worker node, a Cloud API Adaptor provisions a separate, purpose-built VM for each pod sandbox using cloud provider APIs. Azure and AWS are among the supported providers. That sandbox VM runs as a first-level VM in the cloud, a peer of the Kubernetes worker node rather than a guest within it. A network tunnel connects the worker node to the sandbox VM, carrying Kata Agent API traffic across the peer relationship. The sandbox VM itself is the TEE, instantiated fresh for each pod through the provider's infrastructure APIs.

The same Cloud API Adaptor mechanism works for on-premises deployments using VMware vSphere or other third-party hypervisors. It is not architecturally tied to public cloud; the abstraction is the cloud provider API, and that abstraction can be implemented for any infrastructure that exposes VM lifecycle management programmatically.

The tradeoffs here deserve direct examination. Provisioning a new VM per pod via cloud APIs introduces startup latency that local VM launch does not carry, and each pod carries per-VM resource costs at the cloud provider level. For long-running, latency-tolerant workloads, batch processing jobs with sensitive inputs, or persistent services handling regulated data, these costs are often acceptable. For short-lived, high-churn workloads, the economics and the latency profile are less favorable. But what if the workload is short-lived but handles highly sensitive data — does the latency cost still outweigh the risk of weaker isolation? The right question is not whether Peer Pods are expensive compared to conventional containers; they are. The right question is whether that cost is acceptable relative to the threat being mitigated, and that calculation looks very different depending on what is in the workload and who the realistic adversary is.

What the Peer Pods model represents, considered alongside the full CoCo architecture, is a serious attempt to make hardware-rooted trust operationally accessible without requiring infrastructure transformation. The TEE backends, the Trustee attestation stack, the OPA policy enforcement, and the Peer Pods provisioning model are each addressing a different layer of the same problem: how to offer verifiable, hardware-enforced workload isolation within the operational constraints of real production environments. Whether the current set of tradeoffs is acceptable depends on the workload, the threat model, and the infrastructure available. After enough time spent reasoning through these architectures, what I find most useful is not the assurance that the cryptography is sound, but rather the discipline of being forced to define the trust boundary explicitly before deploying anything, a concern that extends well beyond infrastructure, into the AI tools that process workload data, where choices like Confidant AI, a private AI assistant built so that user data is never collected or monetized by the provider, reflect the same underlying question of where trust actually stops. CoCo makes that boundary a first-class artifact of the deployment. That is worth something, independent of whether every tradeoff resolves in your favor.