Prompt Injection Attacks and Defenses

I came across this term ‘Prompt Injection’ for the first time when we released the jailbreaking defense paper back in February. Someone on Hacker News had confused prompt injection with jailbreaking and posted a discussion thread on Hacker News about this paper. After diving deeper into the research, I realized they’re actually quite different things – prompt injection is more like a technique or method you use to attack LLM applications, while jailbreaking is one of the goals you might be trying to achieve with that attack. So, before introducing more works on prompt injection attacks and defenses, I’ll first introduce its definition and discuss about its connection and difference between jailbreaking, which is a far more