If I told you, "I can write a Python program to repeatedly run a machine learning model and calculate statistics over the weights and activations to find and fix a specific problem", would you be surprised by that? Because this is very much a part of what ML engineers do.

And then the experiment where it writes out the weights, in that one it's also writing a Python program to solve the problem --- which, again, is a program a person could write.

It's not nothing that Astra was able to perform these coding tasks, but they're still just coding tasks. They have nothing much to do with the experiment where it reads in the raw weights. In that experiment, the relationship between the input weights and the input is mostly linear, so it's a lot like doing image recognition. It's a cool result but it couldn't work the same way when you have complex transformations happening.

It's exactly what the post says. I was waiting for my excitement to drop for the original research, but I concluded that I was mislead more than this "coverage" than the weight editing research, which was in opinion quite easy to grasp.