> LLMs are dumb and unpredictable, run them enough times and at least one instance will do something stupid and dangerous.

This. The concerning thing is how many engineers do not understand the implications