Yeah, but this is like saying that one needn't focus so much on LLMs making mistakes because humans also mistakes.

They do, but the shape of the way LLMs will confidently mislead you is quite different to the way misinformed humans, or even the malevolent and mendacious humans, will mislead you.