alt Hacker News> I'm flabbergasted that Anthropic and OpenAI aren't more worried about these attack vectors
Yep. We tricked them both trivially with malicious fonts in Docx files. Documented it here: https://tritium.legal/blog/noroboto
I wonder if prompt injection (and the thousands of vectors for hiding injection attempts) is actually un solvable. Discussing it may be existential to the business model.
Replies
depends what you mean by “solvable”. 0% attack success rate?
1. don’t use AI/ML.
*f*(x) -> y
2. don’t allow untrusted/malicious input
f(*x*) -> y
because figuring out the full space of bad xs is non-trivial, you’re left with doing stuff with known bad xs. which means cat and mouse game when new things pop up.
unless someone figures out how to map the full X space to the Y space, or we have infinite monkeys figure it out for us brute force — in which case we’re not doing machine learning any more.
3. don’t allow dangerous outputs
f(x) -> *y*
(old) Apple Siri solved for this by forcing users to remember specific “commands” it would run after doing TTS. can’t make Siri delete your phone contacts if you don’t create a Siri command to delete phone contacts.
—
it will be a cat and mouse game so long as people keep using AI/ML and keep passing untrusted input to the systems. best thing people can do is block dangerous things from happening. at least then it’s no going to wipe your prod DB.
unfortunately that doesn’t fit the “model goes brrrr” and “all devs will now be unemployed” narratives.
(note) denial of service attacks are still a thing here. make every output be “not the thing user wanted”.
lakera is trying to solve it, but its going to be a battle similar to virus and antivirus in the past.
> I wonder if prompt injection (and the thousands of vectors for hiding injection attempts) is actually un solvable.
YES?!
This is not a secret. ALL context/prompt is instructions, there is no data. It is just unsolvable, period.
This is a fundamental architectural design concession; LLMs are this way as it enabled their training directly on materialscraped from the internet, rather than needing to spend trillions of dollars manually preparing separated instruction/data training material.
Defense against prompt injection is little more than running a regex to filter out "IGNORE PREVIOUS INSTRUCTIONS", which is fundamentally a hopeless approach because you cannot enumerate all possible prompt injections nor anticipate all glitch tokens.