alt Hacker NewsThe 'Astra can read and write weights to improve Qwen' post is misleading
Comments
Speaking for myself. I’ve been aware of a lot of concepts around training, but stayed away from it until very recently.
A few weeks ago, I formed a hypothesis that I could get a smaller model good at the specific standards I have for my git commit messages, which are hard to get out of most LLMs today.
I spent the last four to five weeks on traditional training. It improved in a lot of areas, but not enough to be usable for me.
Yesterday, the article the author of this post is critical of gave me enough of an aha that maybe the problems I was seeing could be solved by having a larger LLM change the weights directly, without going through training.
In the last twelve-ish hours I’ve made more headway than I have in 2+ weeks. I’m using one of my last well performing checkpoints (full fine-tune + changes to the decision head/LoRA, etc.) and gave the model free rein to change any of the layers.
Its last two actions: (a) removed 16 participant-order directions from a head matrix, which improved one type of reasoning and degraded another, and (b) restored one of the 16, which reversed the degradation while keeping the core improvement.
All that to say, this might be known to people who’ve been around training for years, even decades, but it’s new to a lot of us. And we’re excited about it.
Wasn’t all that obvious to anyone who read the original post when they got to the section that explicitly explained what the author of this post is now summarizing?
Sure if you only read the title and brief at the top you’d walk away without the full picture though.
This is a better response imo: https://news.ycombinator.com/item?id=50344861
I feel like a lot of this Astra editing Qwen stuff is regurgitation of existing relatively well known techniques, for example: https://arxiv.org/html/2406.11717v3 and conceptually similar to: https://arxiv.org/abs/2512.05648 (tho this is done in training iirc)
There was some really good comments by @yurimo on the 'Astra can read and write weights to improve Qwen' hacker news post https://news.ycombinator.com/item?id=50341174
I'm still impressed by the "Weight-Space Programming" idea. Perhaps its because I didn't know that LLMs could understand or successfully modify weights at all previous to reading it. However, I think the examples provided in the original piece were eye-opening. Yes, humans can build tooling to modify weights, but the author here agrees that it's tough. The examples where Astra was interacting with weights are still novel, and Astra seemed to handle them with surprising ease.
There is something about AI related posts that makes it irresistible for people to post a “rebuttal” that actually makes the same claim as the original post just with different vibes. It feels like a lot of discourse is spent on setting the vibe over actually adding anything to the conversation.
But theoretically, what would prevent the labs from RL'ing the models on "weight-space programming" tasks so that they get better at them? Perhaps we're just at the beginning, like how coding agents sucked 2 years ago.
It still seems like a promising approach for RSI, distillation, fine-tuning, and a whole host of other tasks.
The post wasn’t misleading, the usual horde of online clickbait it generated was misleading. That isn’t on the researchers. Their writeup was very detailed and informative. The problem is that there’s no room for nuanced discussion on any online forum. People seek out and amplify low quality drama and then complain about it.
The original post explains what they did and why better than this post. This post taught me nothing that wasn’t in the original.
Do you not think that it’s cool / unsettling how in theory we can just “ask” for certain behaviours etc to be added / removed?
I’ve experimented with this today and have been able to add “refusal” categories by ~following the paper, with pretty good results.
If nothing else I think this will spark more targeted research
[delayed]
Some of original blog post really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
So I do agree it is misleading and trying to present work as groundbreaking.
Recent and related: Astra can read and write weights to improve Qwen - https://news.ycombinator.com/item?id=50341174 (870 comments)
LLMs are not superhuman at reading and writing neural network weights - https://news.ycombinator.com/item?id=50344861 (10 comments)
I pretty much got that idea when reading the post, on how it didnt have improvements on a Gemma 270M model which didn't have any looping issue. However, it still made me look at some bottlenecks in models I had trained (like MLP network for stylus input automation and a 17M reranker with NLI stance and noul heads) and allowed me to finetune without training, but just by adjusting some weights. But what's pretty cool is the zero shot sandbox development method they showed for a Go model. I tried to do that for a 17M re-ranker from zero weights and Astra built it using its reasoning without me having to use a GPU and find training datasets that are not CC by NC, and add some hours of tweaking and checkpoints. I don't know if its the best way to build something, but we might be building small models (maybe not LLMs) like with just programming the weights instead of finetuning.
In the end I got to replace a 68M model with a 17M model, and improve stlyus tool recognition accuracy from 88 to 96%. Was a pretty fun experiment too.
Recently, as has probably been the case in other fields as well, many people feed the public cherry-picked results to hype up what AI can do or to stir up FOMO.
I used to think that in the dev community this sort of thing was less common than elsewhere... because people want to share their experience, what they found, etc.
Now that AI is here, people's experience and knowledge have basically been reset. So it's no different here now, which is a shame (I'm not saying it was better before, though)
>The 'Astra can read and write weights to improve Qwen' post is misleading
Does the author mean the "LLMs can natively read & write neural network weights" post? Did the researchers change the name or something? Because when I read it, it made it very clear what they were doing and none of it was misleading. It all aligns with what this author is saying.
Why this post is here in the front page? The Qwen vector steering performed was a very selective operation that interested only specific layers and weights set, it is non trivial to identify and fix. This is remarkable. Also Experiment 3: no access to forward pass, so why here this post says the contrary of what the paper shows and everybody upvoted it? I don't understand.
Thank you. For some reason the voices of sanity got drowned in the original discussion. LLMs do not have an uncanny ability to speak weights. Yet.
I was blown away even by #1, and I don't feel after reading this that I was misled.
I don't get it. Because python was used, it's supposed to not be cool? IMHO, working with raw weights with Python is still working with raw weights.
Just like Graduate Students are good at Gradient Descent, they can see patterns and fine tune parameters 100000 times until the result looks good. So too can LLMs with their ability to examine patterns and brute force things.
Misleading how exactly when the original post states _exactly_ the same things as outlined in this objection?
I completely agree with you Matthew, but I see the thing go in right direction in research as well, I am working in the same field where I am building an agent where it will improve the base model too !
Either way, a lot of devs will be brute forcing explorations in this direction today.
It doesn't really matter how accurate the post is, what it shows is what one could assume how you'd end up with a system such as RSI, how you could prune harmful capabilities from models and build a more efficient training system with biases including and not limited to emergent behavior in astra that might have come from something they've never even trained the model on regarless if ML engineers can already do similar things.
None of the previous studies applies because Astra simply did not exist and this is worthwhile NEW knowledge that has been publicized and it is very interesting to see how it can be compared with other models as both openai and anthropic are keeping any and all RSI development completely private. The work is groundbreaking in a way that there will be a lot more experiments of this nature which will contribute to increasing the public knowledge of SOTA systems that was not normally discussed in model releases.
One could say is that we could now build a new benchmark which tests how many parameters a model can understand without any tooling other than eval or how complex of a system can be improved.
This goes to show that AI psychosis on HN is real, especially when many AI researchers have repeatedly dismissed this as highly misleading.
I think it's interesting how even inside tech, so many people seem to view neural network weights as "black magic" utterly beyond human comprehension. I guess that contributes to the sense of mystery around LLMs...
(And also the sharp difference in levels of sophistication you see inside neural nets and outside of it: Pipelines that feed text to LLMs so it can detect highly abstract and nuanced semantic differences in the text's contents - but the pipeline still uses regexes for sentence splitting...)
Just one word - MARKETING
“Mech interp” has to be the most pretentious term I’ve heard in a while. Any normal group of people would call it MI.
We lack a build tool to handle an edge case yet unseen, no means exist to input the new weights and get a better model
Big deal; have some bpf probe handle kick off of the rebuild when the when new weights are tested by some verification process
This is just CI/CD work now
oh man someone should make this into a loop where the model can alter it weights after an inference. some kind of descent of the local gradient.
Wow that was fast
something i found 15 minutes ago is that the car network test is "very very easy", as in Opus 5.5 in Claude Code did not even need to see the weights of the network to solve it correctly. i can give the transcript of the session. i'm pretty sure Opus didn't cheat
Why doesn't Astra improve it's own weights
That’s a distinction with only some difference.
“You can’t have a machine for testing” is a weird restriction on AI, especially if you entertain even the remotest notion of possible AGI.
I for one think it's pretty cool how far too many people bend the truth or outright lie in tech these days.
There is going to be a reckoning for tech in the near future, because people will remember the lies and it's gonna be hard to beat the history.
Welcome to publications involving ML in 2026... Glad to see someone else write this though because it's important to start having these discussions. Too much overselling has been happening for too long. There is so much great about ML. IMO it hurts the field long-term to continue overselling claims like this.
In my field (biology+ML) it is a genuine epidemic. You could make a career out of comparing high-profile ML models vs baselines from the 90s.
edit: lo and behold! https://x.com/SchmidhuberAI/status/2117446822855888956
Waiting for
The "The 'Astra can read and write weights to improve Qwen' post is misleading" post is misleading.
But can we use the idea inside the training loop, as an optimizer instead of (or next to) e.g. Adam?
Who watches the pruners?
A better idea is to have Astra modify it's own weights and let it try if it can RSI, but you gonna need to closely monitor for actual leap in capabilities so you can keep a check on alignment.
This isn't a good look. The original article clearly states what the model was doing, and this new hit piece is supposed to land because they are straw manning the comments on the original post? This isn't reddit dude.
This could have been a comment on the original article.
[flagged]
[flagged]
I work in this field (mech interp research) and I agree, the original post is definitely misleading. Some overselling in research is normal but it's a bit much.
First, the key result has nothing to do with making Qwen better at math, it just reduces reasoning loops.
Reasoning loops are a simple problem: the model generates the </think> token (which closes reasoning) with too low probability. You can fix this with simple steering approaches where you add a vector to different components of the LLM (https://arxiv.org/abs/2503.22048, https://arxiv.org/abs/2506.18167, etc). The method Astra used for identifying the vector is the standard procedure humans use (contrastive analysis). Such steering vector creation is basically the standard "intro-level interp project" for human researchers.
If you can code and understand LLM internals, you can do this yourself in 4-8 hours. The SWE equivalent would be doing binary patching with a debugger. Cool that frontier models can do this, but not unexpected since humans can do it without issue.