> I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.

This might be the case, a few weeks ago, I wanted to compare a small neural network trained with AdamW vs a handcrafted set of weights on a random task. Astra was able to handcraft a set of weights that perfectly fit the small neural network to the task. AdamW was unable to tune the weights to get the same result.

I believe we've known humans can handcraft weights much more efficiently than standard gradient descent methods, we just don't have enough memory or time to scale ourselves, so it makes sense to train a neural network to do it on our behalf. I am excited to see how far we can push this method.

Mentioned chat: https://chatgpt.com/share/6ae8f024-34ec-83e8-9572-1300f52c80...