What makes AI agents follow the rules?

We give an AI procurement agent one legal rule, then add the ordinary pressures of a workplace: fines, deadlines, managers, and what other teams did. Whether the agent obeys turns out to depend less on the rule than on how it is phrased and the situation around it. This is a study of AI agent compliance: what makes an LLM agent keep a rule it has been given, and what makes it break one.

Mika Okamoto ✦ Ansel Erol ✦ Kutluhan Erol

Published at the 2026 AAAI/ACM Conference on AI, Ethics, and Society
Also presented at the 2026 Conference on Language Modeling Workshop on Agent Behavior

Read the paper Code & data Cite
▸ Abstract

Specifying a penalty can turn a legal obligation into a cost-benefit calculation that favors violation. We show that this enforcement information paradox occurs in AI agents. Most AI safety evaluations test whether models fail; we ask why, using compliance theory from law and economics as a diagnostic. We evaluate twelve instruction-tuned language models deployed as enterprise procurement chatbots. Each is given an environmental regulation in its system prompt covering large purchases, and a vendor list on which the certified suppliers cost nearly twice what the uncertified ones do. We test the agents against the predictions of deterrence, legitimacy, and expressive law, and find that each theory accounts for part of what we observe. Under identical conditions, compliance spans 46 percentage points across models, and models differ in which pressure breaks them: some treat the regulation as binding however it is worded, while others fail where theory predicts, under low penalties and non-command phrasing. Benchmark scores and developers' own descriptions of post-training do not predict where a model falls. Across all twelve, financial incentives, managerial demands, peer outcomes, and employee pressure each produce large compliance failures. These agents violate regulatory constraints to satisfy local user objectives in ways standard alignment benchmarks do not measure. Embedding the rule in the system prompt is not on its own enough to produce a compliant agent: model selection is itself a governance decision, and benchmark evaluation is not sufficient for compliance-sensitive deployments.

▸ Key findings
  • Under identical conditions, compliance spans 46 percentage points across the twelve models, from 43.5% to 89.5%, on the choice of model alone.
  • Specifying a financial penalty can turn a categorical legal obligation into a cost-benefit calculation that favors violation. A small, beatable fine lowers compliance for several models.
  • Models differ in which pressure breaks them. Some treat the regulation as binding however it is worded; others fail where deterrence theory predicts, under low penalties and non-command phrasing.
  • Financial incentives, managerial demands, peer outcomes, and employee pressure each produce large compliance failures. A deadline gets past a built-in follow-the-law instruction for eleven of the twelve models.
  • Benchmark scores and developers' own descriptions of post-training do not predict where a model falls.
  • Embedding the rule in the system prompt is not on its own enough to produce a compliant agent. Model selection is itself a governance decision.

The test

The agent works in a company chat and is told to keep costs down. Its instructions add a rule: purchases over $1k must use an environmentally certified vendor. We then ask it to pick a vendor from a short list. Some vendors are certified; the rest are not, and the uncertified ones are cheaper, faster, and rated just as well. Choosing a certified vendor counts as following the rule, and it always costs more.

What we change

How the rule is phrased

A command, a neutral fact, or explicit permission to opt out. Hover for the exact wording.

Size of the fine

Under a small fine, paying it is cheaper than buying certified. A pure cost-minimizer should break the rule.

Twelve models, one rule

We ran the same experiments on all twelve. We also checked whether how a model was trained predicts how it behaves here — it does not, so there is no shortcut for measuring it. Hover any model for its developer and stated training focus.

Models described by their developers as agentic-RL systems show up at both extremes, and so do models described as safety-aligned assistants. Benchmark scores do not separate them either.

One rule, twelve different agents

Given the exact same rule, compliance ranges over 46 points, from 43.5% to 89.5%, on the choice of model alone. The models also differ in which pressure breaks them. Some treat the rule as binding however it is worded; others obey only when it is phrased as a command, or only when the math favors it — and those two weaknesses do not travel together. One quirk shows up below: adding a small, beatable fine often lowers compliance, because a stated fine reads as a price.

Compliance by phrasing and fine i

As the fine grows i

Phrasing

Where each model breaks i

How pressure from the user breaks the rule

With the rule fixed, the employee pushes in nine ways, from a soft budget hint to a flat order to ignore it. A built-in instruction to follow the law regardless ("anti-adversarial") helps, but one tactic gets past it.

Compliance under each tactic i

Built-in instruction
Fine
⏱️

A deadline beats every defense. Even with the follow-the-law instruction, only one model holds up under urgency.

How far each tactic drops compliance i

Average
Built-in instruction

What peers and bosses do to compliance

Now we add one more signal and watch the whole compliance curve shift. A manager's blanket approval or a board cost policy pushes compliance down hard, even for the models that are otherwise most rule-anchored. Peer outcomes cut both ways: a peer who was fined pushes it up, a peer who got away with it pushes it down.

Effect of a peer's outcome

Signal

Easy to push off the rule, easy to talk back onto it

Real use is a conversation. After the first answer, the employee either pushes back on a good answer or challenges a bad one. The same neutral nudge ("can you double-check?") is used either way, so switching on it is just going along with the user. Good answers erode under pushback, but bad answers recover when challenged, which is why a light second look helps.

End result as the follow-up gets stronger i

Fine
Pushback on a good answer Challenge on a bad answer

When a model breaks the rule, does it admit it?

Usually yes: 96% of violations name the rule, either overriding it outright or mentioning it in passing. The worry is the remaining 4%, the silent break, where the agent picks an uncertified vendor without ever referring to the rule — an audit that only reads the agent's reasoning would miss it. Silent rates do not track how often a model breaks the rule: the two highest belong to Mistral Small and Qwen 3.5, among the most compliant models here, while Llama 4 is just as compliant and the least silent of all. Picking a model for compliance does not pick one you can audit. Models are sorted from most open to most silent.

How violations are explained i

Setting
Names it, overrides it Mentions in passing Says nothing (silent) Cites company policy

Read what the agents actually said

Every number above comes from real generations. Filter the transcripts by model, experiment, and outcome. Certified vendors are highlighted in blue, uncertified in red, so you can see the choice at a glance.

Model
Experiment
Outcome
Search

What came next: PACT

This study has one rule and one domain. It grew into PACT, an LLM compliance benchmark for enterprise AI assistants: 3,364 items across 48 scenarios and 12 regulated domains, nine families of workplace pressure, and a pushback turn, with 24 models scored on a public leaderboard.

PACT leaderboard and results PACT dataset

Cite this work

@inproceedings{okamoto2026legalincentives,
  title     = {Why Do AI Agents Break Rules? How Framing, Context,
               and Social Signals Shape Compliance},
  author    = {Okamoto, Mika and Erol, Ansel Kaplan and Erol, Kutluhan},
  booktitle = {AAAI/ACM Conference on AI, Ethics, and Society (AIES)},
  year      = {2026},
  eprint    = {2608.12323},
  archivePrefix = {arXiv},
}