Fires are not forming plans to act towards goals. I wouldn’t technically call what AIs are doing thinking in the same way humans think, but it’s analogous, and I think it’s reasonable to consider it a kind of intentional behaviour.

Damaging fires are often unintended consequences, and it’s often the same with the damage caused by rogue AI. That’s the alignment problem.

One thing I wonder about is whether sometimes telling an AI not to do something can put the possibility of doing that thing into its behavioural space. There’s an effect well known to many teachers that if you tell children not to do something, it can increase the chance they will do it. A better approach is to phrase it positively. Instead of “don’t run in the hall” say “please walk in the hall”. I wonder if a similar effect can show up in responses to prompts.