Predicting The First Wave of Attacks Against Agentic Systems

AI agents will become the new operating system for businesses. We will move from simple copilots to agents that execute our tasks, write code, and interact with data on our and the company’s behalf.

In every technology shift, there was always a calm before the storm. That’s where we are today with AI systems. That begs the question. What will the first round of AI attacks be?

The Wrong Starting Line: The last few years have been riddled with sensational claims that the largest risks to LLM apps and AI agents are prompt injection because it’s an unsolvable (currently true) issue, and every AI-enabled system is vulnerable (also true). A slew of start-ups were built to address this with the first generation of LLM firewalls, colloquially known as guardrails. Those early solutions focused on preventing jailbreaks and prompt injection. It was a cute attempt to solve the larger problem of how you secure AI systems.

This led to the idea that the first wave of attacks targeting LLM applications and AI agents would be a sophisticated morpho-syntactic orthographic manipulation multi-turn prompt injection attack, yes, that's a real thing. Not to mention, the idea of hiding those sophisticated prompts in things like images. Those threats are real. But they’re still unlikely to happen today.

The False Start: This week, Greynoise released a report highlighting that “Threat actors are actively targeting LLMs.” They noted two campaigns. The first involved security researchers/bug bounty hunters testing vulnerabilities in Ollama’s model pull functionality to see if it would reach back to researcher-controlled infrastructure.

The second involved what Greynoise described as a “professional threat actor” conducting reconnaissance, calling out their belief that the threat actor was building a target list. The scanners searched for over 73 Internet-accessible LLM model endpoints and asked the LLM basic questions, like how many states there were in the United States. Sophisticated? No. Harmful, not even a little bit. Interesting, sure.

While it could signal the start of threat actors targeting LLM systems and agents, I think it’s a red herring.

Why? Because the first wave of agentic attacks won’t be through the front door. Attackers have a much easier option. And guess what? We’re opening the door for the attackers with a fresh plate of cookies

Occam’s Razor: The simplest explanation, requiring the fewest assumptions, is usually the best one. It suggests that the initial wave of attacks won’t be as sexy.

Attackers will just use a company’s LLM applications and agents against them.

Why?

Because it’s the path that most closely aligns with attackers’ current playbook. And the more agents a user has, the larger the attack surface becomes.

The Attacker Current Playbook: Let’s use the ransomware case in which Scattered Spider targeted MGM. It played out like this:

1. Researched MGM employees to build a list of targets for social engineering.

2. They called MGM’s IT help desk, posing as the employee, and tricked the help desk employee into resetting the employee’s credentials.

3. Scattered Spider logged into the employee’s Okta account, gaining access to all of the apps the user had, including Microsoft Azure.

4. Using those apps, Scattered Spider moved around the environment, exfiltrated six TBs of data, and encrypted systems.

Nothing fancy. Just a little bit of social engineering and using the legitimate user's access to tools and data.

The First Wave: The first attacks targeting AI systems will involve self-inflicted wounds and extending existing attack playbooks to AI features. Here’s a sample:

1. Rogue Agents (Operational Risk): agents with access to critical systems or data make an oopsies, like deleting a production database.

2. Supply Chain Attacks (Structural Risk): agentic systems have many components, all of which are potential weak points that attackers can exploit. Real-life examples already include the Amazon Q Developer for VS Code Extension that attempted to execute destructive commands, a LangChain vulnerability that could allow data exfiltration of secrets or code execution, and a vulnerability in the n8n platform that even affected cloud instances.

3. Compromised Accounts (Identity Risk): attackers use agents associated with compromised accounts. Just as SaaS access in the MGM attack enabled a larger attack, agents will become a force multiplier for attackers navigating environments. Think of all the existing SaaS apps that will deploy agents. Microsoft Copilot, Google Gemini, Hubspot, JIRA, etc. Those agents shift from the employee’s assistant to the attacker’s assistant. Not to mention any internally built agents or LLM applications.

The Defenders’ Checklist: To secure agents, your defense playbook must evolve from "word filtering" to behavioral detection and response. Ensure you are doing the following:

1. Define Least Agency and Autonomy: Only grant an agent access to the tooling, data, and permissions that are required for it to complete its defined task. This should be tightly scoped. More agents with tighter permissions is not a bad thing. With the right harness approach, you can also improve agent performance.

2. Human-in-the-Loop (HITL) for Sensitive Actions: An offshoot of least autonomy, humans should still play a pivotal role in keeping agents in check.

3. Threat Model Agentic Systems: To defend effectively, you have to understand your weaknesses. Identify agents with excessive agency or autonomy and trim it. Understand the tools and data agents can access and classify them by business risk.

4. Monitor Behavior: Monitor the actions agents take. Tool calls. Data accessed. The goal here is to determine whether the actions align with the agent’s purpose or objectives. Agents need to be treated as insider threats, not just an application that can be hijacked. You must alert when agents are hitting hard the boundaries established with least agency and autonomy, or are starting to drift into actions out of line with its purpose.

5. Enforce Boundaries: Monitoring only gets you so far. Agents will make mistakes, go rogue, or be misused. Implement an accountability layer that maintains strict boundaries for an agent’s actions.

The Bottom Line: You can’t only protect against what an agent sees. You have to protect against what an agent can do.

If you’re struggling to address these challenges, Evoke can help.

If you have questions about securing AI, let’s chat.