I'm not sold on that idea yet.

I don't just have LLMs spit out code. I have them spit out code and then I try that code out myself - sometimes via reviewing it and automated tests, sometimes just by using it and confirming it does the right thing.

That upgrades the code to a status of generated and verified. That's a lot more valuable than code that's just generated but hasn't been verified.

If I throw it all away every time I want to make a change I'm also discarding that valuable verification work. I'd rather keep code that I know works!