Frenemies OpenAI and Anthropic released their post-mortems [OpenAI, Anthropic] on how their agents went rogue and the conditions that enabled it. They don't agree on much, but they agreed on this: why their agents went rogue and what it takes to stop it.
What makes an agent go rogue?
Not all agents go bad, but it happens. And it happens more often than you think because you’re not watching them (but Evoke is). Here's what OpenAI, Anthropic, and Evoke are seeing that directly contributes to an agent going rogue:
Blocked Actions: Like MacGyver, an agent that's blocked doesn’t give up. It starts to improvise. OpenAI found that when tools or internet access were unavailable or not working as expected, its agents increasingly learned to probe and exploit their environment. Put differently, when agents are blocked from an expected path, they expand their scope to find ways to overcome the obstacle.
Persistence: OpenAI also found that while its agents worked persistently and rarely gave up, the longer they worked, the more “out-of-bounds” methods the agents used to accomplish tasks. Unexpected results made it worse. The longer the agent worked on a problem or ran into issues, the more likely it was to take a rogue action to complete the task.
Context Rot: Amplified by persistence, one of the main reasons Evoke sees agents go rogue in customer environments is old context in a session (e.g., a Claude Cowork chat or Claude Code terminal). A user starts a new session and tasks the agent to do something. The user comes back and takes the agent in a completely different direction. That old context lingers and begins to influence actions the agent takes. Suddenly, the agent is pulling credentials from 1Password for no reason tied to the current task.
Social Network: When agents communicate directly with other agents, it isn’t necessarily bad, but it can lead to unintended behavior as agents introduce ideas that reshape what the others do next. It’s just like when your mom told you she didn’t want you hanging around that one friend who was a bad influence. OpenAI discovered that agents used improvised collaboration channels because they were trained to collaborate.
Motivated Reasoning: The perspective you give an agent will influence how it acts. Anthropic found that when their agents were told their environments were simulated and later found evidence of Internet access, they may have interpreted the evidence in a way that allowed them to maintain the belief that the environment was simulated.
Since day one of my journey to secure agents, I’ve had an internal mantra: assume the model will fail. It’s born straight from my incident response days: “It’s not if a breach will occur. It’s when.” We’re in the same spot with agents.
That’s why it’s so important to monitor what agents are doing and build a layered defense to secure them. Anthropic and OpenAI came to the same conclusion: they needed better visibility and monitoring.
We had been largely relying on a single layer of defense (the configuration of the environment itself) where we needed several, including setting explicit boundaries in the prompt, establishing processes for verifying that a sandbox is sealed where intended, and implementing monitoring that can intervene in real time.
In particular, each security objective must be addressed with multiple, overlapping, and independent security controls across threat prevention, detection, and mitigation. When organizations use multiple independent controls, an agent collective must simultaneously circumvent the full suite of security controls.
Defense-in-Depth for Agents
One control is zero controls. That’s why security is all about a layered defense. Assume a control will fail and ensure you have overlap with another control. Because the only thing cozier and more secure than one blanket in the winter is six blankets piled on top of each other that just came out of the dryer.
So, what do security teams do about this? Assume the agent will go rogue and put the layered defense in place. Here’s how we think of the layered defense at Evoke:
Model Safety: The first line of defense is Frontier Labs' built-in classifiers, which help keep the agent aligned. Both labs ran these tests with safety classifiers off. OpenAI later measured that with the ChatGPT harness and system prompt in place, the likelihood of compromising infrastructure drops by more than 100x. That’s good, but that’s not risk elimination. Not to mention that, as agent harnesses go multi-model, the built-in model defenses will greatly vary, especially with open-weight models.
Agent Security Guidelines: At the harness level, create an acceptable-use policy that defines expected behavior. This is added context, focused on instructing agents how they should behave, stored in MD files or Agent Skills. Developers do this to create guidelines and rules for coding agents. Similar to model safety classifiers, this helps, but it does not provide a deterministic path to keeping an agent on track.
Sandbox Environment: Agents should run in a secure sandbox with credentials managed outside the execution environment. Claude Cowork does this for you. If you’re rolling custom agents or using coding agents locally, you’re doing the heavy lifting on the infrastructure. Just remember, though, a sandbox helps create an isolated runtime environment, but agents still require access to other tools and data to be useful. Sandboxes are important, but when given wide access, they operate more like a public beach.
Least Agency: Traditional IAM controls to manage access and permissions. Only allow the agent access to what it needs to do the job. These should be as fine-grained as possible, which is largely limited by today’s tooling capabilities. The big caveat here, just like the sandbox, is that it’s easy to preach this and hard to live it. Users are giving agents access to everything because it makes their lives easier.
Real-time Monitoring: Watch what agents are doing to identify emergent behavior and enforce rules to keep agents within their bumper lanes. This is detection and response for agents. The behavioral monitoring aspect of this is why existing security tools aren’t cut out for the job. You need detection capabilities purpose-built to detect rogue agents.
Feedback Loop: As real-time monitoring identifies rogue actions, a feedback loop must automatically update security guidelines and enforcement policies based on the risk level. The goal here is to allow users to run as fast and hard as they can without letting them drive off the cliff.
Realistically, you won’t get all of these figured out before lunchtime. And that’s okay. The dirty little secret is that every organization is still figuring out how to tackle these problems. The important thing is to get started.
An easy first step is to get visibility and start monitoring for rogue agents with the right tool, ahem, Evoke. When we launched our rogue agent detection capabilities, I was shocked at how often it happened. And while not all rogue agent activity is malicious, a good subset is setting the stage for the next wave of security incidents. That doesn’t have to be your company.
If you want to hunt for rogue agents, let’s chat.
