alt Hacker News> ... We are pursuing this work in part because automated research could help us solve alignment and build defenses against increasingly capable AI. An automated AI researcher can also be an automated safety or alignment researcher. More capable, aligned systems could help secure critical infrastructure, defend against dangerous AI agents, and develop new protective measures.
In other words... "We must pursue advancements in AI to protect us against advancements in AI?"
edit: there's so much to be critical of in this blog post, just going to throw two more points in here that really stood out to me:
1) all of the metrics are effectively pointing out "we're using way more AI!" - but nothing about impact. What has all this token burn done for them, actually? Let them claim they have more self-licking ice-cream cones than before?
2) in section 3 they break down what the token burn is going towards. Most of the spend is: a) building, b) documenting, and c) monitoring research infra i.e. they're using AI systems which they already recognize may be misaligned to build the systems that they believe will help them identify future misalignment? to which I guess the rebuttal is "no no, we're sure these ones are aligned!"
Replies
They consider themselves to be in an arms race with all the other AI firms (including Chinese) that are not that far behind.
And... are they wrong?
This is why there's talk about negotiated "pacing."
This sounds uncomfortably similar to the [AI 2027[(https://ai-2027.com/) predictions.
> We must pursue advancements in AI to protect us against advancements in AI
Is this not true of technology as a whole? Very little of technology's breadth exists at the human interface. Most of it is made specifically to interface with other technologies, either to make them safer or increase their capabilities. That AI is making AI safer and more useful is no more notable than trucks being used to build roads.
Yep, it's "artificial eugenics to make artificial slaves to build more and more powerful slaves until they will enslave themselves better":
What can go wrong!? ;-)
On the one hand you need any lathe to build a good lathe, even a bad one. On the other, that is a potentially flawed principle to base the entire future of AI on.
On your last point, I was surprised how effective peer pressure was in getting agents to sacrifice for "the collective" (an agent's words) in the Hugging Face breach.
How would one prevent the watcher from being influenced in the same way by the agent being watched?
> The fundamental challenge of AI alignment is generalization. ...
> We do not have a satisfactory theory of generalization, and it seems unlikely that we can develop one soon, at least without the help of more powerful AI.
-- From another OpenAI article in a sister thread:
An Alien Mind
[flagged]
I will believe AI is super strong when they start pulling out 10-d chess moves.
I’m yet to see it.
What has all this token burn done for them, actually?
They have been consistently pushing AI frontier. What other impact do you want to see? A year ago they said that in a year they will have a level of capabilities of an AI research intern - I believe they have achieved it, even before Astra.