alt Hacker NewsAstra can read and write weights to improve Qwen
Comments
Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."
OK, so is this how LLMs will be able to learn?
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?
2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.
Such an unexpected result...
Completely shook. Is this the start of it? At the very least one (not sure if the first?) example of strictly superhuman capabilities.
Is it aware of projects like [1] and implementing those?
I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
Wow, in a way it's like CRISPR for ML models!
>Qwen3.5-0.8B
I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.
The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.
GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.
This raises so many questions.
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
Curious to know if models other than Astra can do that.
Is there any chance they’ve just seen these weights before and are recognising them?
If not this is mind blowing.
However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.
That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
Throw out whatever timeline you previously had
Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.
Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next
This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
Wild
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
Friendship ended with AdamW. Astra take the wheel.
But for real this is insane. Okay I need to get on this stat.
Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
RSI
I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.
This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).
I wonder how it would do with distilling models by hand?
Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.
Great, instead of backpropagation, we now have slopagation.
Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?
This is wild. Could Astra read and potentially audit a mapped human brain in that case?
So how would this be in the training? Does OpenAI train models on their own epochs?
Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?
(When they say "Writing a Go algorithm," they mean the board game, not the language.)
Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?
ok, that's a bit scary..
Did they try it with other frontier LLMs? What about other non-trivial out of distribution tasks? So many questions.
... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...
this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
This doesn’t look like “normal” technology to me…
I wonder if this behavior emerges when using recursive self improvement to generate new models
"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."
At what cost? That's the part I'm curious about.
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?
Mind blown.
I simply must know now: Could they do it with neurons in a brain?
Some absolutely crazy and insane experiments if this ability grows.
Could they edit an animal's brain? Can they teach things to animals???
I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster
Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
That's how you get to rsi, a surgeon operating his own brain.
So, i guess editing other model weights will be banned tomorrow
this is maybe one of the most bonkers machine learning research i've seen all year.
Whaaaaat!? This is crazy right?
I'm just blown away. Editing weights? Writing weights from scratch!?
almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest