Zero-Trust Architecture for AI Inference Infrastructure

Zero-trust is not a product. It is also no mere checkbox. After years of watching organizations conflate "we deployed an identity provider" with "we implemented zero-trust," I've grown impatient with how casually the term gets used. The distinction is architectural, not semantic, and it matters most in exactly the environments where the confusion is most common: distributed, multi-cloud AI inference deployments where no single perimeter contains the workload.

NIST SP 800-207, published in August 2020, established the U.S. federal normative baseline: zero-trust moves defenses from static network perimeters to continuous verification applied to users, assets, and resources. The more directly relevant document for AI infrastructure is NIST SP 800-207A, published in September 2023, which extends the framework to cloud-native and multi-cloud environments. It introduces service-mesh architectures, sidecar-proxy enforcement, and SPIFFE-based workload identity as reference implementation patterns. Those specifics are not incidental; they are the difference between a conceptual framework and an implementable one.

Four principles from those documents recur throughout this analysis. Never Trust, Always Verify means every request is authenticated and authorized regardless of origin, including requests from services that already cleared a perimeter check. Assume Breach means treating compromise as statistically probable rather than hypothetical, and designing fine-grained controls accordingly. Least Privilege means access rights restricted to the minimum necessary for the intended function, not the minimum someone bothered to configure. Microsegmentation means isolated segments with specific policies that prevent lateral movement after an initial compromise.

The CISA Zero Trust Maturity Model v2.0, published in April 2023, organizes these principles into five pillars: Identity, Devices, Networks, Applications and Workloads, and Data. Each pillar has a direct analog in the AI inference stack.

Regulatory context sharpens the urgency. U.S. Executive Order 14028, signed in May 2021, required federal agencies to adopt zero-trust architectures by September 2024, with AI systems named as an explicit focus. Enterprise practitioners outside the federal sphere have occasionally treated this as someone else's problem. That reading is harder to sustain as the frameworks converge with the actual incident record.

The conceptual shift that matters most: zero-trust for AI is not primarily about who logs in. It is about applying identity, policy, and verification to every workload, every service call, and every data retrieval in the inference stack, including calls made by automated processes with no human in the loop.

The Concrete Incidents That Show What Inference-Layer Exposure Looks Like

In April 2024, Wiz Research demonstrated the first publicly documented cross-tenant compromise in an AI-as-a-Service environment. Researchers uploaded a PyTorch model containing a malicious reduce method to a shared inference environment. When the serving container loaded the file, arbitrary code executed. Cross-tenant model files and authentication tokens were readable from inside the compromised container. The vulnerability was not in the model's weights. It was in the serving infrastructure, the deserialization behavior of the framework, and the absence of tenant isolation at the workload level.

The serialization problem is not isolated to that incident. In December 2024, JFrog disclosed 22 vulnerabilities across MLflow, H2O, PyTorch, and MLeap. Roughly half shared the same root pattern: model file formats that execute native code on load. The attack surface is the artifact pipeline, not the training loop.

CVE-2025-32434, assigned a CVSS score of 9.3 and disclosed in April 2025, made the situation considerably more difficult. This vulnerability broke torch.load(weights_only=True), the canonical "safe loading" mitigation that the PyTorch ecosystem had recommended as the defensive posture against deserialization attacks. It affected PyTorch 2.5.1 and earlier. A single CVE invalidated a community-wide defensive assumption. Patching quickly resolves some exposure, but when the mitigation itself is the vulnerability, the community's entire defensive posture needs rethinking, not just its patch cadence.

The model-theft literature adds a different dimension. A survey covering published research from 2021 through 2024 found that model extraction attack success rates ranged from roughly 34% to 99%, depending on query budget and target architecture complexity (arXiv:2605.29450). The breadth of that range is itself the point. It spans from "meaningful risk" to "near-certain success under favorable conditions." Neither end of that range supports the conclusion that inference endpoints are adequately protected by perimeter controls alone.

On the prompt-injection front, OWASP's Top 10 for LLM Applications, published in November 2024, placed prompt injection at the number-one position for the second consecutive edition. OpenAI has publicly acknowledged the problem is unlikely to be fully resolved at the model level. The UK's National Cyber Security Centre has characterized it as potentially more severe than SQL injection, given the degree of trust modern applications place in model outputs.

What these incidents share: they all exploit the inference layer, not the model itself. Supply-chain vulnerabilities in serving frameworks, cross-tenant isolation failures, unsafe deserialization at load time. The perimeter, in each case, was not the relevant boundary.

The organizational cost is not abstract. In 2025, 84% of organizations experienced identity-related breaches. The average cost of a breach reached $5.2 million, 38% higher for organizations that had yet to implement zero-trust. The inference layer is where that cost materializes.

Why Workload Identity Is the Foundation Everything Else Depends on

Every control that follows in this analysis depends on the ability to answer one question with cryptographic certainty: which workload is making this request, right now? Without that answer, authorization policies are built on assumptions, and assumptions in security work the way they do everywhere else: they hold until they do not.

The inference stack creates identity problems that human-user identity systems were not designed to handle. Training pipelines, preprocessing jobs, model-serving pods, and evaluation runs each require distinct, verifiable identities, separate from the humans who authored or deployed them. The default in most organizations is static secrets: API keys, long-lived tokens, shared service accounts. Those defaults are also the failure mode present in most of the incidents described above.

SPIFFE (Secure Production Identity Framework for Everyone) and its reference implementation SPIRE address this directly. Rather than treating identity as a static credential issued once and rotated rarely, SPIFFE turns identity into a short-lived, cryptographically attested property of the workload itself. SVIDs, SPIFFE Verifiable Identity Documents, are typically valid for one hour, configurable to minutes, and rotated automatically by the SPIRE agent before expiry. A compromised pod cannot impersonate a different service. It holds only its own SVID, scoped to its own service account, valid for a window measured in minutes. The blast radius of credential theft shrinks by an order of magnitude.

Production deployments at Uber, Stripe, and Netflix as of 2025 validate the operational viability of SPIFFE/SPIRE at scale. NIST SP 800-207A names SPIFFE-based identity as a reference implementation pattern, not a theoretical option.

Mutual TLS between model instances adds a second layer: transport-level authentication that ensures no service can receive a request from an unauthenticated peer, and that generates an audit trail of which model version served which request. That trail is what makes post-incident reconstruction possible rather than speculative.

There is an AI-specific identity gap worth naming explicitly. Unlike APIs, AI agents are probabilistic systems. Two replicas of the "same" agent will behave differently across sessions. From a security and compliance perspective, this means every agent instance requires a unique identity, not a shared service account. Shared service accounts were already a poor practice for stateless APIs; they are architecturally wrong for probabilistic systems that can take multi-step autonomous actions against enterprise systems.

How Microsegmentation Contains the Blast Radius When an Inference Endpoint Is Compromised

The premise of microsegmentation is not that compromise will never happen. The premise is that when it happens, the damage should be structurally contained. That is the "Assume Breach" principle applied to network design, and it is distinct from, not a substitute for, continuous verification. Both are necessary; one without the other is a partial answer.

Without segmentation, a compromised inference endpoint has a clear path to training data stores, feature catalogs, conversation history, and enterprise system integrations. The inference container needs some of those resources to function; it does not need all of them, all the time, without restriction.

A practical four-zone model for AI systems organizes the landscape usefully. Zone 1 covers AI inference infrastructure: model API calls, response caching. Zone 2 covers RAG retrieval infrastructure: vector databases, document stores. Zone 3 covers conversation history and user-specific data. Zone 4 covers enterprise system integrations, the CRMs, ITSMs, and ERPs that agents increasingly interact with. Each zone boundary has explicit firewall rules; no zone trusts another zone's traffic implicitly.

The separation between training and inference deserves particular attention because the two workloads have nearly opposite network profiles. Training requires large data inflows: model weights, feature sets, training corpora. Inference requires minimal data access: query inputs, retrieval results, cached responses. A compromised inference endpoint should be architecturally incapable of reaching training data. That gap is not just a network rule; it is a deliberate design decision that prevents data exfiltration via model manipulation. Feature stores, which necessarily bridge both worlds, warrant column-level access control so that inference endpoints access only the specific features they need, not the full catalog.

The multi-cloud reality complicates this. The majority of organizations now operate in hybrid or multi-cloud environments; segmentation policies that exist only within a single cloud provider's networking constructs offer incomplete protection. This is the operational context NIST SP 800-207A addresses with service-mesh enforcement: policies that travel with the workload across environments rather than depending on a fixed perimeter.

Policy alone will not catch everything. Runtime detection tools, tuned with AI-specific rules, provide the complementary layer. Unexpected network connections from model-serving pods, file-system access to training data from inference containers, anomalous GPU memory usage patterns: these are signals that policy enforcement may have been bypassed. They require active detection to surface.

Continuous Verification at the Data Access Layer, Including RAG Retrieval

The default failure mode in RAG deployments is straightforward and surprisingly common: permissions are evaluated at index time, when documents are ingested into the vector store, rather than at retrieval time, when a user actually requests information. A user who loses access to a document after indexing can still receive its contents through a RAG-generated response. The access control system says no; the retrieval system says yes; the user gets the data.

This happens because retrieval and access control were typically designed by different teams, integrated under time pressure, and the gap between them was left unguarded. I have seen this in organizations with otherwise mature security practices, which suggests it is a structural problem, not a negligence problem.

Zero-trust data access for RAG addresses this at the retrieval step itself. Each retrieval query generates a fresh access decision. Document-level permissions are enforced at retrieval time, not pre-indexed. Retrieval results are filtered to the requesting user's current authorization level before they are passed to the model for synthesis. This is not a novel architectural concept; it is standard access control logic applied to a system where it has frequently been omitted.

Session-level continuous verification adds a temporal dimension that static permission checks miss. Trust should not remain static across an extended conversation, particularly when agents are handling multi-step tasks with access to enterprise systems. Re-authentication thresholds tied to session duration and to the sensitivity of requested actions provide a practical mechanism: a session that began with a low-sensitivity query and escalates to requesting write access to an ERP system is a different security context and should be treated as one.

Least-privilege enforcement at the serving layer is a structural boundary, not just a policy setting. Training jobs receive read-only dataset access, with write permissions limited to designated output locations. Model-serving endpoints expose inference APIs only; they lack training capabilities or direct data-access permissions. These should be separate workload identities with separate credentials, not separate roles assigned to a shared account.

The API gateway is the enforcement point where retrieval-layer policy and network segmentation converge. No AI component should have direct network access to enterprise systems. All such access passes through a gateway with authorization controls. That chokepoint is where the policies described in this section become observable and auditable rather than assumed.

Policy-as-Code as the Mechanism That Makes Zero-Trust Consistent Across Environments

Manual policy management and inference infrastructure are incompatible at production scale. Model versions change frequently. Serving containers are updated, replicated, and replaced. New integrations are added. Each change is an opportunity for a policy gap to open, either because the new configuration was not covered by existing rules or because the existing rules were not applied to the new component. In practice, gaps accumulate quietly; they surface during incidents.

Policy-as-code resolves the consistency problem by treating security policies as versioned artifacts that travel through the same deployment pipeline as infrastructure definitions. Policies defined in code, deployed via tools like Terraform alongside AI infrastructure, and enforced via Kubernetes operators in containerized environments are reviewable, diffable, and auditable. They are not undocumented manual configurations that exist in someone's memory and drift silently between environments.

The composition principle that connects the components: deny by default, allow specifically, log every decision. SPIRE provides workload identity; the service mesh enforces network policy; policy-as-code defines authorization rules. The "log every decision" clause is what makes the system observable. Every inference request, which workload, which model version, which data it accessed, generates a traceable record. Incident response becomes investigation rather than reconstruction.

The AI-specific addition to standard policy-as-code practice concerns the deployment pipeline itself. Policies should govern how models access data and external services during inference, not merely how human operators deploy models. A malicious or compromised model artifact that can be promoted to production through an ungated pipeline, and then reach enterprise data stores without policy checks, represents a supply-chain attack path that policy-as-code is positioned to close. Model promotion gates that enforce access-control policies as a condition of deployment are the mechanism.

Hardware-Rooted Trust and Confidential Computing at the GPU Level

Software-layer controls assume the execution environment is trustworthy. Hardware-level attacks dissolve that assumption.

When a large language model runs inference, the computationally intensive work happens on the GPU: weight matrices multiplied, activations computed, outputs generated. If a Trusted Execution Environment protects only the CPU, an attacker with access to GPU memory can extract model weights, intermediate activations, or inference prompts regardless of how well the software stack is configured. Organizations that have invested heavily in software-layer zero-trust controls should find this clarifying: those controls do not extend to the accelerator unless hardware attestation is explicitly included.

GPU-level confidential computing extends the TEE boundary to include the accelerator. Weights and intermediate computations are encrypted on-device. Hardware attestation provides cryptographic proof that the execution environment is authentic and untampered before any sensitive data or proprietary model is loaded into it. "Never trust, always verify" applied to silicon: the serving environment must prove its integrity before it receives anything worth protecting.

NVIDIA's H100, based on the Hopper architecture and shipping in July 2023 with general availability in 2024, was the first GPU to support confidential computing. The Blackwell architecture, announced in March 2024, extended GPU-CC further, including encryption of traffic between GPUs collaborating on large model inference. The Vera Rubin NVL72 extends this to rack scale, protecting GPU execution, memory, and register states across 72 Rubin GPUs and 36 Vera CPUs simultaneously, keeping models, training data, and inference prompts isolated at the infrastructure level.

The justification for hardware-rooted trust is not theoretical. Research has demonstrated GPU timing attacks achieving a 100% success rate under controlled conditions. That figure is a demonstrated attack class against the class of hardware that serves AI inference, not a worst-case edge case. Hardware attestation is the answer, not a supplement to it.

What a Phased Implementation Looks Like for Organizations Starting from a Traditional Perimeter Model

Most organizations are not starting from zero. They have perimeter security, some identity controls, possibly a VPN and a firewall that segment broad network zones. What they typically lack is workload identity for AI components, per-query access control at the data layer, microsegmentation specific to AI workloads, and policies that travel across cloud environments. The gap is not between nothing and zero-trust; it is between coarse-grained perimeter controls and the fine-grained, continuous verification that inference infrastructure requires. That distinction matters because it changes where the work actually starts.

Phase 1: Inventory and Workload Identity

Before deploying any control, map the actual topology. Identify every inference endpoint, every data source it touches, and every inter-service communication path. This inventory will surface assumptions that do not survive contact with the actual architecture, and it is the prerequisite for every subsequent phase.

Deploy SPIFFE/SPIRE or a comparable cryptographic workload identity system before adding other controls. Identity is not one control among many; it is the precondition for every authorization decision that follows. Enforce mTLS between model-serving components as part of this phase. The combination provides cryptographically verified identity and transport-layer authentication before any data-access or segmentation policy is applied.

Phase 2: Segmentation and Least-Privilege Data Access

Implement the four-zone network model. Separate inference GPU clusters from training GPU clusters at the network level, not just at the policy level. Enforce column-level access at feature stores so that inference endpoints retrieve only the features their workload identity is authorized to use. Move RAG retrieval to per-query access decisions, eliminating the index-time permission gap.

Deploy an API gateway as the single authorized path from AI components to enterprise systems. Every request from an inference component to an external system passes through that gateway; nothing reaches enterprise integrations directly.

Phase 3: Policy-as-Code and Continuous Verification

Codify all policies accumulated in Phases 1 and 2. Integrate them into deployment pipelines so that new model versions inherit controls automatically rather than requiring manual reconfiguration. Add model promotion gates that enforce access-control policies as a condition of deployment.

Add session-level re-authentication thresholds for AI agents, tied to session duration and to escalating action sensitivity. Deploy runtime detection with AI-specific rules: unexpected network connections from serving pods, file-system access patterns inconsistent with the workload's declared function, anomalous resource usage.

The architecture that emerges from these three phases is not a finished state. The inference stack will continue to evolve; attack techniques will continue to develop; new hardware capabilities will create new attestation possibilities. What this sequence establishes is the foundation: verified identity, bounded network topology, continuous policy enforcement, and observable decision logs. Those properties do not make the problem disappear. They make it manageable, auditable, and incrementally improvable rather than opaque until the next incident.

Where, specifically, the gap between current controls and this model is widest in a given organization: that is where the next incident is most likely to originate. That is the more productive question to be asking right now.