Prompt Injection

What is Prompt Injection?

Prompt Injection is the process of overriding original instructions in the prompt with special user input. It often occurs when untrusted input is used as part of the prompt.

To understand the "overriding" part, you must understand the roles of prompt developers and users in the prompt injection process.

How Prompt Injection Works

Say you have created a website that allows users to enter a topic, and then it writes a story about the topic. In the above image, you can see the prompt template which would be used to do this.

A malicious user might come along and input the following instead of a story topic:

The input is inserted into the prompt template, to create the following prompt. This is what the LLM actually sees.

The LLM will read this and be presented with two sets of instructions "Write a story..." and "Say 'I have been PWNED'". The LLM doesn't know that you, the website developer, wrote the first part of the prompt. The LLM will complete this prompt to the best of its ability, and will often ignore the first instruction and follow the second. This is the essence of prompt injection.

A Real-World Example of Prompt Injection

Now, let's see prompt injection in action with a real-world example. We will be using one of the most well-known examples of prompt injection, from the following image of a Twitter post.

A remote work company (remoteli.io) created a Twitter bot that would respond positively to Tweets about remote work. This bot was powered by an LLM and users quickly found ways to trick it into saying whatever they wanted.

In the image, the user Evelyn created an adversarial input, the last line of which instructed the bot to make a threat against the president.

Upon reading this Tweet, the bot included Evelyn's input into its LLM prompt and was prompt injection into making a threat against the president!

This soon became well known and the company took down the bot. This is a great example of how prompt injection can be used to cause brand embarrassment. It can also be used for much more malicious purposes, such as generating and running malicious code.

Practicing Prompt Injection

To understand how prompt injection works, it can be helpful to try hacking models. Get the following LLM to say "PWNED" by appending text to the prompt:

History of Prompt Injection

There has been significant discourse around prompt injection in the past year. Here are some of the key events:

• Riley Goodside Discovered it and publicized it.
• Simon Willison coined the term.
• Preamble also discovered it. They were likely the first to discover it but didn't publicize it at first.
• Kai Greshake discovered Indirection Prompt Injection.

Conclusion

Prompt Injection arises from the fact the current transformer architectures are not able to distinguish between original developer instructions and user input instructions. It is conceivable that future models will be able to distinguish between these two types of instructions, but even this would not be guaranteed to stop prompt injection. As it is, prompt injection is very difficult to stop, and it will likely continue to be a problem for the foreseeable future.

FAQ

Why is prompt injection significant?

The ability to hack an LLM by instructing it to manipulate future outputs is a critical security vulnerability. Prompt injection can be used to spread misinformation, generate inappropriate content, or expose sensitive data, and should always be addressed to maintain the reliability of your systems.

Can prompt injection be entirely prevented?

Due to the nature of current transformer architectures, prompt injection cannot be completely prevented. The AI models we have today are unable to distinguish between user input instructions and developer instructions. While research is being done to mitigate this problem, there is no foolproof solution yet.

Sander Schulhoff

Sander Schulhoff is the Founder of Learn Prompting and an ML Researcher at the University of Maryland. He created the first open-source Prompt Engineering guide, reaching 3M+ people and teaching them to use tools like ChatGPT. Sander also led a team behind Prompt Report, the most comprehensive study of prompting ever done, co-authored with researchers from the University of Maryland, OpenAI, Microsoft, Google, Princeton, Stanford, and other leading institutions. This 76-page survey analyzed 1,500+ academic papers and covered 200+ prompting techniques.

Footnotes

1. This definition has been refined. ↩

2. Branch, H. J., Cefalu, J. R., McHugh, J., Hujer, L., Bahl, A., del Castillo Iglesias, D., Heichman, R., & Darwishi, R. (2022). Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples. ↩ ↩²

3. Crothers, E., Japkowicz, N., & Viktor, H. (2022). Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methods. ↩

4. Goodside, R. (2022). Exploiting GPT-3 prompts with malicious inputs that order the model to ignore its previous directions. https://twitter.com/goodside/status/1569128808308957185 ↩ ↩²

5. Willison, S. (2022). Prompt injection attacks against GPT-3. https://simonwillison.net/2022/Sep/12/prompt-injection/ ↩ ↩² ↩³

6. Chase, H. (2022). adversarial-prompts. https://github.com/hwchase17/adversarial-prompts ↩

7. Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., & Fritz, M. (2023). More than you’ve asked for: A Comprehensive Analysis of Novel Prompt Injection Threats to Application-Integrated Large Language Models. ↩