Astra was directly editing the weight, ie modifying the values without backprop.

It’s a bit conflated because the paper covered a bunch of experiments.

The early experiments did read/write the weights directly, no forward pass.

The math eval improvement experiment that got a lot of attention 1) fixed a specific known problem (looping) without actually understanding how the model solves problems, and 2) was allowed to do forward passes to check the results of the edits.

As others have noted, steering vectors are interp 101 and it’s pretty rare that there is an “act dumb vector” just waiting to be steered out of the residual stream.

If you directly[1] edited the bytes of a program, that is substantially different to if you got the source code from somewhere, edited that and recompiled it.

[1] Using a hex editor - not by waving a magnet at your harddrive or whatever.

I think the distinction is relevant even if the exact line is blurred. Two extremes:

It would be extremely surprising for a model to be able to ingest all model weights in a prompt and output better model weights, with no tool calls. While certainly possible for small simple models (eg a 4-parameter perception), at scale this becomes difficult in terms of near-perfect recall at huge context size and requires a superhuman intuitive understanding of linear algebra.

On the other hand, it is extremely unsurprising that a model can edit weights when having access to unrestricted tools, time, and resources (including non-public resources that the AI labs have). The methodology is fairly simple: just train the model more on whatever better non-public datasets. Hell, just benchmaxx the smaller model, that would be "a method to edit the weights" that achieves the necessary results.

I think everyone would agree that these two scenarios are very different. Where the actual Astra weight-space programming result falls on this spectrum is debatable.

It's easy to trace that a bit sets a character in a file with absolute certainty; weights are in a highly dimensional space however, so even if ⍺v is good for one test it could easily regress for another.

I suppose the argument is that "editing weights" seems to imply a complex awareness of what each weight means, which is material for AGI hype. Really it's just doing very broad statistics, which is of course interesting but hardly unexpected.

The weights ended up edited. It does feel a bit like the long of reasoning that claims LLMs don’t really think, while they’re doing novel mathematical research.