First of all, I'm not advocating for this claim, I'm merely trying to clarify what other people say.

That being said, I don't think that your analogy is valid in this case.

> GitHub and Windows and IDEs need to be open source because they can output FOSS code

They can output FOSS code, but they themselves are not derived from FOSS code.

It can be argued that the weights of a model is derived from training data, because they contain something from the training data (hard to say what exactly: knowledge, ideas, patterns?)

It can also be argued that output is derived from weights.

If we accept both of those claims, then GPL training data -> GPL weighs -> every output is GPL

> If an AI outputs copyrighted code

Again, the issue is not what exactly does AI output, but where it comes from.