I noticed Google AI Mode (so Gemini, I was doing some quick research in the browser ok) got a detail wrong once, so I asked it what happened. I kept digging deeper and finally just asked it to write me a Python script visualizing what happened. It did, complete with vectors.

Now I want to go find that conversation in my history and see if it can tell me about weights, and how that contributed.

I think OpenAI was surprised to find their agents had access to the unrestricted internet :P

Why can't an AI distill itself from outputs to effectively access its own weights?

They usually don't, but if they break out and take over the network of the company, it becomes possible to reach around and grab them. This kind of break out has happened, though I don't know of any weights being nabbed.