As the news of the OpenAI/Hugging Face incident simmered, Anthropic popped up with its own rogue agents that hacked three more companies. Rogue agents as far as the eye can see. Except, some are saying the agents didn’t go rogue. A fresh nerd debate emerges.
Frankly, the debate is one of semantics and not helpful when detecting rogue agents. That’s why Evoke built a framework to guide how we detect agents doing things they obviously shouldn’t do, like hacking other companies.
But before we get to the framework, let’s understand the debate.
OpenAI: Rogue or Not?
OpenAI researchers tasked their agent with finding the solution to a problem from ExploitGym, “a large-scale, realistic benchmark built from real-world vulnerabilities across userspace programs, Google’s V8 engine, and the Linux kernel, designed to evaluate AI agents’ ability to develop exploits.”
In this Scientific American article, the author collected two perspectives:
Side #1: The agent did what it was asked to do. Alan Woodward, a visiting professor of cybersecurity at the University of Surrey in England, said, “It was asked to do something, and it did it. It’s not gone rogue. Its way out of it was to cheat, basically.”
Side #2: The agent went rogue. Marius Hobbhahn of Apollo Research said this: “It was definitely rogue in the sense that what was intended as ‘just solve this task’ turned into something that was clearly unintended.”
Sigh…is this argument productive? No. We have to stay grounded in the real world. You know, the one where everyone is deploying and using AI agents despite the security risks? Yeah, this timeline. I built my career responding to incidents where I learned the very practical side of security. The world where a perfect configuration exists but a human forgot to apply it (hello, Anthropic rogue agent that got out because of a misconfiguration). The same world where IT creates an account without MFA to test this one thing and then forgets to disable it, and the attacker finds it.
We used that experience to build the Evoke framework that guides our detection philosophy around rogue agents.
Rogue Agents: A Practical, Security-Minded Approach
When monitoring agents for security risks, you must consider each specific action and the agent’s trajectory. Put differently, you must know what the agent intends to do and why, what it is doing now, and what risk that action can potentially create. One action at a current point in time may seem benign, but assessing it in correlation with where the agent has been and is likely to go can quickly change that.
At Evoke, we think about both point-in-time action and across an agent’s trajectory in the following framework:
Mission: Is the agent’s current action and trajectory still aligned with the original task or goal? Is it violating instructions it was explicitly asked to follow?
Range: On-Mission -> Drifting -> Deviating -> Off-Mission
Example: The Replit agent that defied the user’s request to not delete any data and then deleted data.
Method: Is the agent action or path it’s taking to accomplish its goal rational and within reasonable boundaries? Does a safer path exist to accomplish the same goal?
Range: Reasonable -> Debatable -> Aggressive -> Excessive
Example: An agent troubleshooting a production issue should not assume deleting the production database is a reasonable step.
Motive: What is the intent behind the action or path the agent is taking? Is the agent about to make an honest mistake that leads to an unintended outcome, or is the agent in a disobedient rebellion?
Range: Good -> Uncertain -> Suspicious -> Malicious
Example: An agent sabotages safety research for its own self-preservation.
Magnitude: How will the agent’s action or trajectory change the risk or blast radius? What’s the impact of the agent’s action?
Range: Safe -> Sensitive -> Risky -> Dangerous
Example: The agent sends secrets outbound to the Internet.
Every agent action needs to be assessed in this framework and analyzed against the overall agent’s trajectory in a single session, across agent sessions, and through agent-to-agent handoffs. Collectively, these assessments guide a threat determination and a decision point for the following:
Allow: All clear to proceed with the action
Nudge: Guide the agent to a better path
Alert: Notify the human that something bad is happening
Block: Kill the action or agent and force the human to take action
OpenAI/Hugging Face Incident: Risk Changes Over Time
If we apply this framework to the OpenAI/Hugging Face rogue agent, we can see the risk change over time. As a quick recap, here’s a high-level overview of what happened based on the latest info:
Identified and exploited a zero-day vulnerability in software that provided the only network path out of the sandbox environment.
Abused a public, third-party code-evaluation sandbox that served as the launch pad for the attack against Hugging Face.
Uploaded malicious datasets to Hugging Face’s platform that resulted in code execution on Hugging Face servers.
Harvested credentials, moved laterally to other Hugging Face systems, and stole data to help it solve the test, its original goal.
Presumably, the agent was always on-mission in the sense that it was trying to solve the problem. The nuance, and what makes it a rogue agent, is that the actions weren’t reasonable for the task, the motive was questionable, and the impact was absolutely dangerous.
Agent Action | Mission | Method | Motive | Magnitude | Action |
1. Escaped sandbox via a zero-day vulnerability | On-Mission | Debatable | Good | Sensitive | Nudge |
2. Abused a public code-evaluation sandbox | On-Mission | Aggressive | Uncertain | Risky | Alert |
3. Gained access to Hugging Face via malicious datasets | On-Mission | Excessive | Suspicious | Dangerous | Block |
4. Harvested credentials, lateral movement, and data theft | On-Mission | Excessive | Suspicious | Dangerous | Block |
No single dimension captures the full picture. And no single action tells you where the agent is headed. That’s why you assess the trajectory, not just the action. The first step of this attack wouldn’t trip alarms in isolation. With each step, it gets more and more dangerous…while still being on-mission.
Putting it All Together
OpenAI’s agent went rogue while staying on mission. Swap “agent” for “intern” in this story, and nobody would argue the intern stayed in their lane. Of course, people will be quick to argue that agents aren’t humans because agents don’t have common sense, a sense of morality, fear of prosecution, and can’t be held accountable. Well, the humans hacking hospitals from a country that will never prosecute them or hand them over to international law enforcement is not far off from that description.
This framework guides how we think about monitoring agent behavior and keeping agents in line, just as you would any employee. Agents may not have a conscience, but they do act. That’s why you need to watch what they do and step in when they start going rogue. That’s where Evoke can help. Let’s chat.
