I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.

They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.

LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.

The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.

Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.

So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.

The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)

So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.

As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into

I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?

It sounds impressive at first but after you think it becomes mundane.

OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.

It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.

My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.

EDIT: I think the bigger implication is that backprop is not the only game in town.

"No human can go edit a brain to become better at math" -- I'd offer this is exactly what "learning math" is.

I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.

I am not convinced a human could not edit another human's "weights" to improve their performance if such a thing were possible and could be done safely and could be reversed -- all of which are true in virtual neural networks like these.

E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.

What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.

I agree this is a huge deal, but I dont think this is saying that the NN can read the weights and "say hmmmm this one looks a bit low its obviously causing reasoning loops, lets bump that up a bit". The LLM that modified the weights developed hypotheses and wrote a bunch of code to look at various mathematical properties of the weights and updated them based on that. While it's still crazy it's not quite what I was imagining when I first read this...

Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.

> No human can go edit a brain to become better at math

You could have done what was reported.

This reminds me what professor Geoffrey Hinton did talking many times in the past. Super intelligent artificial aliens.

I think the most interesting take is to look at their thinking logs to find out how exactly they decide what weight to change and why.

How are the weights even being mapped into token space as input? Are they bypassing tokenization so the weights just come in as what must seem like gibberish? Are they stringified as numerals then tokenized?

Why is it so surprising? Humans can make compilers that are better coders than they themselves are.

LLMs are exposing how much untapped potential comes from something that isn't necessarily more "intelligent" than humans, but is simply magnitudes faster at reading and analyzing data.

I have so many questions.

Could a model then transmit concepts to another?

Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?

What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?

Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.

I’m more shocked by how many commenters in this thread seem to be misunderstanding what is going on here.

I am very shocked by this as well, although it also kinda feels like it makes sense, you know what I mean?

If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!

Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.

its a program optimising a program it has nothing to do with a human reprogramming their own brain

So I just asked Astra if it thinks it could do this and it said no, it can't.

Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?

during all the openai agent sandbox shenanigans, did it manage to access it's own weights?

> No human can go edit a brain to become better at math.

For now...

I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.

i think one thing to take from this is that "brains are not much like neural networks beyond some superficial concepts"

I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.

This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud.

Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.

We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.

Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.

It's not to store and copy around versions of C and Python and JS. They don't exist organically.

This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]

[1] https://arxiv.org/abs/2309.10668

[2] https://arxiv.org/pdf/2605.27734

Another bad day for the stochastic parrot theory.

I mean, pattern recognition and manipulation is what these models are ideal for.

It is a very cool discovery, but I'm not surprised that you can do it.

[dead]

All the labs are training these models on LLMs/machine learning like crazy.

Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.

This just seems like a more automated, faster version of what humans have been doing with open-source models.

It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.