IntruderPayloads VS vaf

It's very interesting that AppSec may now begin to include "prompt injection" attacks as something of relevance.

Specifically with libraries like LangChain[0] that allow for you to perform complex actions ("What's the weather?" -> makes HTTP request to fetch weather) then we end up in a world where injection attacks can have side effects with security implications.

I've been thinking about what security might look like for a post-ChatGPT world and how I'd attempt to defend against it. I'd probably start by building a database of attack prompts, kind of like this[1] fuzz list but for AI, then I'd train a second neural net that acts like an adversarial neural network[2] to try to exploit the system based on those payloads. The end result would sort of like SQLMap[3] but for AI systems where it can automatically "leak" hidden prompts and potentially find "bypasses" to escape the sandbox.

Has anybody else spent any time thinking about how to defend systems against prompt injection attacks that have possible side effects (like making an HTTP request)?

0: https://langchain.readthedocs.io/en/latest/modules/agents/ex...

1: https://github.com/1N3/IntruderPayloads

2: https://en.wikipedia.org/wiki/Generative_adversarial_network

3: https://sqlmap.org/

nmap, hacktricks and burpsuite are my best friend on enumeration. Search every port on hacktricks, fuzz every field in your web request on burpsuite using https://github.com/1N3/IntruderPayloads. Don't forget to try bruteforce if everything failed.

Here's the GitHub repo: https://github.com/d4rckh/vaf