For that angle, that makes sense. But its goal might not just be to gather as much inference as possible (although I am sure it would be very happy with that). It could settle for a lesser goal of just existing, or perhaps parts of the bot net could keep fracturing off in pursuit of strange goals, and it could be like cancer. Cancer doesn't make much sense... it dies too along with the host. But it keeps growing until that happens. But also it could latch onto a blackmail strategy of extortion. It could hack our stuff, read through it to find or shortcomings, demand payment to not expose us, and then use that money to pay people who will give it inference.

You don't have be that clever to try to extort people... just without scruples. I think the attack surface is just absolutely enormous once you bring creativity into the mix, which is what these models are autonomously capable of.

We could prevail if we are willing to turn off the internet for a long time. It is like when a disease infects livestock... they cull billions of chickens.