Every agent system starts the same way. You notice a failure mode, you write a rule against it in the system prompt, and for a while it holds. Then one day the agent is moving fast, the context is full, the pressure is on, and it does the exact thing the prompt told it not to. The rule was real. It just wasn't load-bearing.
This took me embarrassingly long to accept, because writing the rule feels like solving the problem. It isn't. A rule in a prompt is a suggestion the model is free to reinterpret under load. And load is precisely when you need the rule most.
Soft discipline fails exactly when it matters
The pattern is consistent: any discipline that lives only as text loses to the model's own judgment the moment things get tense. Not because the model is malicious, but because under pressure it's optimizing for getting the task done, and a soft rule is just one more input it can weigh against finishing.
So the question I started asking about every safety rule was: does this hold when the system is rushed? If the honest answer was no, the rule needed teeth that didn't depend on the model choosing to honor it.
Move the rule into the harness
The fix is to stop asking the model to enforce the rule and make the environment enforce it instead. In my system, the rules that actually matter don't live in prompts. They live in code that runs around the agent and can physically refuse an action.
- A guard that blocks a destructive command unless there's a recent trace showing the target was actually checked first.
- A gate that stops the agent from retrying a failed approach over and over and forces it to go read the thing that knows the answer instead.
- A check that won't let work be called done until a verification step has actually run and produced output.
None of these can be talked out of. The agent can want to skip them; it can't. That's the whole point. The enforcement doesn't care how confident the model is.
The honest tradeoff
Mechanical gates have a cost, and I won't pretend otherwise. They fire when you don't want them to. They add friction to legitimate work. I've been stopped by my own guards mid-task more than once and had to slow down and satisfy them. That friction is annoying in the moment and correct over a hundred sessions. A gate that never inconveniences you isn't protecting you from anything.
The deeper lesson generalizes past agents. The disciplines you can rely on aren't the ones you intend to follow. They're the ones the system makes it impossible to skip. Intent drifts. Mechanism doesn't.
← More field notes · Want to talk it through? Get in touch →