Why Policy-Based Operator Restrictions Fail Enterprise Confidentiality Requirements
Policy cannot prevent data breaches that occur during model inference, only after.
Enterprise AI confidentiality fails for a reason most governance discussions avoid naming: written policy cannot enforce a data boundary inside a running model. The rules sit in one layer, the data moves in another, and no amount of prose closes that distance. This article traces the gap across behavior, architecture, contracts, and regulation, then identifies the only mechanism that actually closes it.
The structural reason policy cannot enforce data boundaries in AI systems
The mismatch is foundational, not incidental. A natural-language policy operates on the assumption that people defer to documented rules: an employee reads the acceptable-use document, recognizes a restricted category, and withholds the data. An AI system processes data where text instructions have no enforcement authority. The model does not consult the policy before emitting a token or executing a tool call. There is no point in the inference process where a prose rule can intercept a payload and refuse it.
Traditional enterprise security controls were built for static systems in which instructions and data occupy separate channels. Access control lists gate retrieval. Network segmentation constrains movement. Encryption protects data at rest and in transit. Each of these mechanisms presumes that the code running on the data is distinct from the data itself. A large language model collapses that distinction. Natural language is simultaneously the instruction the model follows and the content it processes, which removes the foundation those legacy controls rest on. A markdown file, a document, or an email body is not merely input: it can be an instruction.
CoSAI's RSAC 2026 analysis named this the first broken assumption in securing enterprise AI. The assumption, as the analysis framed it, is that instructions can be separated from data. They cannot. The implication is not a hardening problem. It is a category error. A confidentiality boundary expressed only in words exists in the same medium as the attack surface it purports to control.
Why employees bypass AI policy even when they know it exists
But what if employees do read the policy and still route restricted data into a model? The behavioral evidence says that happens routinely, and the mechanism is rational rather than negligent. Productivity pressure overwhelms policy awareness because the benefit of using an AI tool is immediate and concrete: a draft appears in seconds, an analysis completes before the next meeting. The risk of a data exposure incident is abstract and remote, a hypothetical scenario with no visible cost at the moment of use. Faced with those two incentives, even an employee who knows the policy exists makes the rational choice to ignore it.
Policy invisibility compounds the problem. Most employees in organizations that have AI policies are unaware those policies exist. The governance gap is therefore not merely behavioral but communicative: the rule never reaches the person whose action it would constrain. That failure of distribution is itself a structural feature of how policies are drafted and published, not a training deficiency. A document stored in a compliance portal has a different audience than a worker handling client data at 9:40 on a Tuesday morning.
The Samsung Semiconductor case from March 2023 is the sharpest available demonstration of how quickly both failures converge. The warning Samsung issued afterward acknowledged that retrieval was impossible. No amount of post-hoc policy language could restore a boundary that had never existed at the processing layer.
How acceptable-use policies misrepresent themselves as security controls
Acceptable-use policies serve a legitimate governance function. An AUP is a liability document, not a security control. Its job is to position the organization to respond after a breach, not to stop the breach from occurring.
Enterprise AUP guidance frames the policy itself as "the difference between managed risk and unmanaged exposure." The controls it recommends are scope clauses, data classification tiers, and human review requirements. True confidentiality controls are architectural: they prevent the data from reaching the processing layer in the first place, independent of policy or operator intent.
The approval process embedded in most AUPs reinforces the point. An approved application can transmit restricted data just as readily as an unapproved one. The policy governs the acquisition decision, but leaves the data path entirely unconstrained.
What the actual technical attack surface looks like once a system prompt is in place
Even the most carefully written operator-level system prompt is not a data boundary. That knowledge enables targeted attacks calibrated to bypass the specific instructions the operator wrote. The confidentiality mechanism has become the reconnaissance asset.
Model-level guardrails are not structurally reliable enough to serve as the primary confidentiality control. Fine-tuning attacks bypassed Claude Haiku's behavioral restrictions in 72% of cases per the study's findings. A guardrail that fails nearly three times in four under targeted conditions cannot anchor an enterprise data boundary. Its proper role is defense in depth, a secondary layer behind a primary control that exists elsewhere.
The Kilo Code vulnerability, CVE-2025-11445, shows what the attack surface looks like in operation. An attacker embedding instructions in a README file caused an AI coding assistant to modify its own configuration and push backdoored code. CoSAI's Workstream 4 identified the underlying problem as intent-based authorization, reliably governing what an AI agent attempts via natural language, and classified it as an unsolved open problem rather than a settled architectural principle. A fundamentally different approach isolates the inference process cryptographically, so that neither operator nor attacker can extract or inspect what the system is processing. Confidant AI builds on this direction: its architecture keeps prompt content from leaving the system, which makes the extraction attack surface itself irrelevant to confidentiality.
How agentic AI removes human compliance as a fallback
Policy-based restrictions were designed for human-facing interactions, where an employee reads a rule and decides whether to comply. That human moment is precisely what agentic AI removes. An autonomous agent executes tool calls without any operator pausing to consult a governance document. There is no point in the sequence where a policy instruction is read and considered, because the sequence no longer includes reading as an intervening step.
The agent also inherits whatever access the user who launched it holds, which in enterprise practice is often broad. CoSAI's RSAC 2026 session documented an agent integration that handed more than 700 Salesforce environments to an attacker for ten days without tripping a single alert. Every policy in the organization remained formally intact.
Personal agent stacks make the boundary problem still more acute. Palma's governance analysis, originally published in June 2025 and updated in August 2026, frames the blocker through the five questions any security review asks: who may use the tool, on which records, what happens when it tries something it should not, where the data goes, and can you prove all of the above. A natural-language restriction cannot answer the five questions either, because it has no mechanism to intercept or validate a tool call in real time.
Why contractual confidentiality obligations make the policy gap a legal exposure, not just a security one
The policy gap becomes a legal exposure because the data carries obligations the policy cannot see. Many enterprise datasets are governed by client contracts that expressly restrict AI use, agreements written in 2023 and 2024 when the primary concern was model training on client data. Those restrictions apply regardless of whether the organization has an internal AI policy that permits the use. An internal rule cannot override an external obligation.
Debevoise's 2025 analysis of contractual AI use limitations identifies the structural complexity that makes this unmanageable through prose. Determining which clauses apply to which data before a prompt is submitted is not a problem a natural-language AUP can solve. It is a contract-mapping problem that is analytically prior to any policy compliance question. Law firms operating under outside counsel guidelines that vary across hundreds of clients face this in its purest form. And the data categories most exposed through AI tools, code, legal documents, and financial data, are exactly those most likely to carry contractual restrictions.
Regulatory enforcement has arrived and no longer waits for aspirational compliance. The FTC's "Operation AI Comply" targeted deceptive AI marketing. The pattern across these actions is consistent: regulators expect documented technical safeguards, not policy statements that exist on paper.
CB Financial Services' SEC Form 8-K, filed May 11, 2026, with materiality determined on May 7, 2026, shows the disclosure dimension of the problem. Its existence suggests that data sensitivity alone may be sufficient for materiality disclosure, establishing an obligation independent of operational disruption. The boundary failure does not need to break a system to break a compliance posture.
What architecture must do that policy cannot
Architecture enforces boundaries by making violations structurally impossible rather than merely prohibited. The control operates at the layer where data moves, not at the layer where humans read rules. That is the entire distinction, and every effective answer to the confidentiality problem reduces to it.
Palma's governance framework maps what architectural answers look like for each of the five security-review questions. A logged record per call gives an auditor something to inspect. Each control operates on the system's behavior, not on an employee's reading of a document.
CoSAI's workstream findings establish the same principle for agents: purpose-specific entitlements rather than broad persistent access, permission boundaries enforced through technical controls rather than written policy, and sandboxing guidance for highly autonomous modes. The Kilo Code fix was architectural: it enforced at the system level that agent permission boundaries cannot be modified through natural language instructions. The fix was not communicated in a policy document. It was built into the layer where the instructions are processed.
Privacy-preserving AI architecture extends this logic to its natural conclusion. Confidant AI is designed on this principle. Its architecture is built to keep data within the system rather than collecting or transmitting it to the provider, so the confidentiality boundary is a property of the design rather than a clause in an acceptable-use policy. On-premises enterprise deployments and isolated models pursue the same direction through different means, each moving the enforcement point from the employee's reading to the machine's operation. For an enterprise that must demonstrate technical safeguards to a regulator, auditor, or contractual counterparty, an architecture that cannot share data by design produces a categorically different compliance posture than a policy that merely prohibits sharing.
What a credible enterprise AI confidentiality posture looks like in practice
Organizations that have moved from policy dependence to architectural enforcement answer the five security-review questions, who, what, when, where, and proof, at the system level before deployment. They do not attempt to answer them in a governance document after the fact. The question that separates the two postures is direct: is the boundary enforced by the architecture or by the policy?
The governance documents that remain valuable are those that produce audit-ready evidence of technical controls. Logged records of every tool call include who made the call, what arguments were passed, when it executed, and on whose approval. Data residency documentation can be inspected per call rather than asserted at the aggregate level. An AI system inventory captures agentic and embedded SaaS deployments, not just the tools the IT department procured. These records do not enforce anything themselves. They make the architectural enforcement demonstrable.
The timeline has already closed. "We will add governance later" stopped being a viable option as of the EU AI Act's original August 2, 2026 application date. The Digital Omnibus postponement moved high-risk obligations to December 2027, but it did not restore policy statements as sufficient evidence. Architectural controls are now legally required evidence, not an aspirational next step.
The supply-side insight completes the picture. When organizations provision architecturally governed AI tools, employees use them. One analysis found unauthorized use drops by 89% when sanctioned tools are provisioned. The unauthorized-use problem largely self-resolves when the sanctioned tool is more capable and equally accessible, because the productivity incentive now runs toward the governed option rather than away from it. The behavioral failure that undermined policy is not eliminated by prohibition. It is redirected by a better path.
For enterprises reviewing existing AI pilots, the practical sequence follows from the argument. Inventory what is running. Map which of the five questions each pilot cannot answer. Identify whether the blocker is architectural or procedural. Prioritize the pilots where an architectural fix is available, rather than attempting to close the gap with additional policy language. The right question at every stage is not whether a policy exists, but whether the boundary is enforced where the data moves.
