What about the investors who put money into those companies and want to get some money out of that? Does the law say that the companies owe them? Something I thought about recently...

Often when people describe SI or similar, there is a frame often taken which describes a machine so smart and capable that it becomes uncontrollable. The hypothetical is easy to understand but it requires too many assumptions and an allergy to nuance/complexity to be believable for me.

The more realistic outcome, and in my opinion the more scary argument to not proceed without guardrails is that SI is achievable and is built without its owners and operators losing control: the worlds most powerful, privately owned super weapon that operates as an infinitely capable forgery within the Internet, a plane that we all share and depend on despite its opaque downsides with respect to an inability to verify authenticity.

We didn't need AI for FTC astroturfing to influence regulations back in the net neutrality days. We didn't need AI to disrupt meat space by creating and scheduling a protest and counter protest across the street from one another. We didn't need AI to mold public opinion, even in times when that new form was more distant from the truth.

What made these influence ops difficult to conduct safely (read: without being caught) is what made them rare (relative to today): they are plays of big risk for big reward. But over time social media commoditized it, and in doing that made it easier to do and more centralized, the most glaring example being TikTok and the bipartisan effort to ban it.

Now, buying US phone numbers from startups that run racks of "phones", buying swarms of pre-warmed social media accounts, and other unscrupulous methods of masking inauthentic behavior has become an accepted organ of the VC space. The industries cultural vibe of "fuck you, you can't stop the future" turns criticisms into marketing.

While we get placated with fears of nuclear or AI induced disaster and stories about machines that may now be alive, the psychosis is taking hold which has shifted the conversation away from examining what is happening from the perspective of accountability to a perspective akin to watching a chemical reaction take place.

The noise has created a permission structure to behave in ways that are otherwise unjustifiable. And baked in are the roots for excuses to be made when the inevitable realizations down the line.

In the mean time, we are supposed to be having this public discourse about what is happening and what should happen next. I trust that these AI companies see using their super weapon today, here and now, in order to pave the way to a more secure future down the line.

Sorry that I used your post to soapbox. I agree, the goal is flawed indeed.

Super intelligence is either super cool or super powerful, no way not building if it is a choice. Progress cannot be stopped, those responsible for it may collapse from time to time I guess but someone else will pick it up and go further.

IMHO There's no future where we don't have real human-type artificial intelligence or super intelligence. It will happen simply because it already exist but the production requires humans having sex and looking after the product for decades.

Instead of trying to prevent it, lets look for ways to deal with the dangers of it.

> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

Speak for yourself. A future like in The Culture novels sounds great to me!

> a future where humanity is the pet species of a god-like artificial intelligence is hardly an appealing one!

I strongly feel that points of view on this are going to be almost 100% correlated with standard of living.

Maybe a good option would be to have the 10% of the world with the worst situations - starvation, parents w/ dying children, suffering violence etc - vote on whether we turn things over to the superintelligences. This would incentivize society to make sure the floor is extremely high.

I'd prefer a world where humans don't get overtaken but IMO I don't think it's moral for comfortable citizens to have the final say.

I thought it's just a marketing gimmick, just like how cars should be safely driving by itself by now.

There is no reason to presume a superintelligence we build will want pets, feel antipathy to humans, or be god-like in any way, or even have any goals other than what we give it. If you jump to this conclusion, regardless of how many big words on less-wrong you attach to it, this speaks only about you and not about AI. If you think the super-AI will hate humans, it's because deep inside you hate humans. I also hate humans, but I know that humans are making the AI, it probably will not hate them, in addition to surpassing me in other abilities.

In general, the smarter a person is, the nicer and more helpful they are. There is no reason to presume differently of AI, especially if we are building it to be helpful to ourselves.