alt Hacker NewsIt doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
Replies
How do you prune out the 'fails to prune out behaviors' behavior? If my model is failing to prune it out, can another model prune out that behavior?
> how you could prune harmful capabilities from models and build a more efficient training system
In theory, if OpenAI or Anthropic wanted to release a new open weight model, they could scrutinize it to remove the most horrendous things from the model.
What it shows are two unrelated results mixed together for no apparent reason besides marketing. The raw-weight results are impressive, but haven't been demonstrated as ready to scale up to a real language model. The Qwen result is a mildly interesting application of functionality that's been on the market for years.
If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.
If it isn't accurate, it isn't showing anything. The Qwen math claim is entirely false in its presentation, it unblocked an existing capability. It did not instill a new one which is what an RSI loop would actually have to do
"[...] how you could prune harmful capabilities from models [...]"
We already do this, abliteration is the term that describes the pruning of capabilities or safeguards for models
"[...] not limited to emergent behavior in astra that might have come from something they've never even trained the model [...]"
If I was attempting to build RSI, I would have these exact types of problems in my RL datasets. There is no proof this is emergent behavior.