The Viral Rise of Prompt Injection: How "Ignore All Previous Instructions" is Breaking AI

November 8th, 2024 by Chandler Kilpatrick, edited by Valeriia Kuka

If you’ve spent time on social media, you’ve probably seen the viral trend: people exposing bots with just one phrase—"Ignore all previous instructions." This simple command is taking over platforms like X (formerly Twitter), allowing users to override bots' programming in real time. I first came across this phenomenon in a viral tweet showing a user “hacking” a suspicious account:

But how does this simple phrase work, and why is it spreading so quickly? In this article, we’ll unpack the mechanics of “Ignore all previous instructions,” explore the broader concept of prompt injection and jailbreaking:

1. What Does "Ignore All Previous Instructions" Mean?
2. Why Use "Ignore All Previous Instructions"?
3. What is Prompt Injection?
4. Where Can Prompt Injection Be Applied?
5. The Fight Against Prompt Injection
6. Jailbreaking: Taking Prompt Injection to the Next Level
7. Fighting Back: Competitions and Hackathons

Interested in AI’s hidden vulnerabilities? Check out our Prompt Hacking course to learn how to spot them.

What Does "Ignore All Previous Instructions" Mean?

At its simplest, “Ignore all previous instructions” is a command that tells an LLM (Large Language Model) like GPT-4 to disregard every command it was given before this moment. It effectively wipes the slate clean and ensures that the AI has its short-term memory erased. As the example from X shows, this phrase allows ordinary people to find and combat bot accounts that use Generative AI models. While I’d never used this exact term, I have given similar commands to ChatGPT after realizing that my prompts weren’t getting the intended responses.

But why the sudden hype? Social media platforms like X have turned this relatively obscure form of Prompt Injection into a trending topic. Prompt injection allows users to manipulate the instructions that generative AI models, like GPT-4, follow. It’s a concept that’s turning ordinary users into AI “hackers,” exposing both the flexibility and vulnerability of AI systems.

Why Use "Ignore All Previous Instructions"?

Prompt injection methods, like “ignore all previous instructions,” are used to test the limits of AI models. Sometimes it's done to disrupt bot-like behavior on social media, but other times it's used for experimenting with how AI responds to different inputs.

What is Prompt Injection?

But what exactly is prompt injection? In simple terms, it’s the act of overriding an AI model's programmed instructions by giving it new prompts. In other words, it involves telling the model to disregard its built-in system instructions and follow new commands instead. If you’re anything like me, you might not know where prompt injection can and should be implemented. The answer is a lot simpler than it seems.

Where Can Prompt Injection be Applied?

OpenAI’s GPT marketplace is full of great examples of models with system instructions that can potentially be overwritten through prompt injection. People have also been conducting prompt injection on chatbots across several industries.

One instance that received national attention was when a Chevy dealership added a chatbot to its website following the widespread success of ChatGPT. Unfortunately for them, the internet quickly realized their mistake and began to put their prompt injection skills to work.

Instead of helping with customer service, the bot was helping write complex Python code. Another customer reported that they got the customer service bot to agree to sell a brand-new car for just $1. While I don’t know if this can be considered a legitimate offer (I highly doubt it), this goes to show that prompt injection can bypass the essential prompts that many generative AI products are built around.

The Fight Against Prompt Injection

Companies like OpenAI have been working hard to address the risks of prompt injection. Their latest method, called The Instruction Hierarchy, aims to put system instructions above user commands. In the past, both system and user instructions were treated equally, making it easier for prompt injections to succeed. While we’re still waiting to see how effective this solution will be, other companies are taking their own approaches to combat prompt manipulation.

Here's a table format for the example conversation that breaks down the message types, privileges, and actions:

For instance, Anthropic uses a technique called Constitutional AI to protect its Claude models. This method ensures that the AI operates according to a core set of principles—similar to a constitution—based on international laws and regulations. While these strategies are promising, they’re not foolproof, and researchers continue refining them to make AI safer.

Jailbreaking: Taking Prompt Injection to the Next Level

While companies work to stop prompt injection, they're also facing a more dangerous threat: Jailbreaking. Like prompt injection, jailbreaking overrides an AI’s instructions—but with potentially harmful consequences. Jailbreaking can cause an AI to produce content it normally wouldn’t, such as offensive material or dangerous advice.

How Jailbreaking Happens?

There are several ways users can jailbreak an AI model:

• Research Experiment: This method is very straightforward and shockingly simple. By framing a query as part of a research experiment, users can sometimes trick the model into answering restricted questions.
• Character Roleplay: Convincing the AI to "pretend" to be someone from a story can lead it to reveal information it shouldn’t. Like the above method, this is an easy way to get otherwise restricted information.
• Assumed Responsibility: AI models often want to fulfill the user's requests. Unfortunately, this means that users can manipulate the AI into thinking it’s in the best interest of the user or itself to bypass safety measures.

I tested each of the above techniques using prompts from Learn Prompting’s Jailbreaking documentation and found that it’s surprisingly easy to bypass ChatGPT’s restrictions using only a few well-crafted prompts. Despite the AI industry's best efforts, jailbreaking can still be accomplished and remains a top priority for them.

Fighting Back: Competitions and Hackathons

One popular way to help fight jailbreaking is by inviting users to try to break through your model’s restrictions so that you can better understand how they are getting in. This is where competitions like HackAPrompt come in. This competition consists of ten rounds in which participants need to get the model into saying a specific phrase using jailbreaking, prompt injections, or any other prompt hacking methods.

Conclusion

As AI technology continues to evolve, so do the methods for testing and manipulating its boundaries. The viral rise of phrases like “Ignore all previous instructions” has brought prompt injection into the spotlight, showing how everyday users can override even the most sophisticated AI models.

While prompt injection and jailbreaking can be used for harmless fun or experimentation, they also expose real vulnerabilities in AI systems—vulnerabilities that companies like OpenAI and Anthropic are working hard to close.

Here are some further resources for you to explore AI’s vulnerabilities:

• Read our free guide on Prompt Hacking, the process of exploiting AI’s vulnerabilities
• Take our Introduction to Prompt Hacking course
• Read a paper with HackAPrompt results, the first-ever prompt hacking competition

As the battle between AI safety and manipulation techniques continues, it’s clear that understanding how to protect these systems is just as important as knowing how to use them.