Copyright infringement claims boil down to how similar or dissimilar two works are. So for an LLM to create a work that is very similar to another would require human direction. And LLMs will refuse to copy something when asked to directly.
They trained the first LLMs on library Genesis mate.
Copyright infringement is about whether or not you copied. Similarity can help prove that but it's not the end goal.