50+TFlops is nothing, I got that in my MacBook, but besides that, when, a few years/decades from now, whatever arbitrary compute limit we think prevents Armageddon comes down to enthusiast and consumer level, what then? This isn’t Uranium, compute is not a physical resource.

This is the “SGI” regulation issue I never read a reasonable answer to, if one believes this is possible and should be prevented then either that means they want to restrict every computing system sold from here on out to some arbitrary metric (and somehow prevent users from just creating clusters to get around such a compute restriction) or what?

If compute alone directly leads to “SGI” or whatever, then we might as well put paper bags on our heads and lie down in some English pub.

Not to mention, if one really wanted to cause harm, training a current day LLM and using it for Stuxnet-esque attacks is reasonably possible long before any arbitrary compute limit we might introduce now, no machine God needed to cause major harm.

That’s why I prefer advocacy for LLM regs that focus on current day impact. Mental health concerns, training data licensing questions and the like. There I can formulated reasonable regulation that can hold. For “SGI”, I do not know anyone who actually has done that and I have looked hard. That’s why I consider these things more distraction from actually necessary and possible regulation that just draws attention via a flashy doomsday scenario.

Occasionally, I will click on one of the AI Doomsday Youtube videos recommended to me. And far more often then not, these will posit that "SGI" requires only compute and will inevitably cause devastation. Fair enough, I still think we should put a bit more focus on e.g. LLM induced psychosis, the labs rarely compensating those whose training data they used, etc. but if it is their opinion that "SGI" is possible, I can get why they'd ignore such concerns. But at the end, they never state how to regulate or prevent this, they more often then not have a call to action ("If you want to prevent this...") linking to a website where we can actually read about how they think we should deal with this. Inevitably, I click on said site, finding it to for one be an Effective Altruism aligned project and B always just contain some blabla about "aligning AI training with human values", which is absolutely meaningless nonsense, not least after having watched a video in which someone spends 15 minutes espousing that "we could never fully control "SGI"".

Makes all these feel more like industry efforts to stave of necessary regulation and not actually serious, but if one can formulate how to regulate “SGI” that isn't laughable, nonsense or both, I am not opposed, I just don’t think that person exists…