And if an AI generated those wights, they are not copyrightable
That could effectively be the case here but your statement in isolation is an oversimplification of recent legal guidance on this matter -- AI generated things are copyrightable if there is sufficient human authorship involved in the generation process.
The copyright office's recent statement on this matter were sensationalized when they really said nothing groundbreaking at all -- this was always the standard applied when any tools are used in creating a work.
Even if the weights are just made through traditional training and no AI is used to make the weights... is there a copyright right? If the weights are a numerical derivation that aren't made by a human is there even human expression? Absent human expression, there is no copyright right?