AI is getting pretty good at exploiting vulnerabilities https://news.ycombinator.com/item?id=49990763
And this was foreseen: https://www.anthropic.com/glasswing
All it is going to do is lead to a greater regulation of models that get released.