alt Hacker Newssame thing as people just entering a question prompt and copying and pasting the response as gospel. e.g. politicians using it to write speeches, or lawyers for testimonials.
the output is programmed to look correct so unless you have some sort of background you won't actually know what errors to look for.
And look correct is accurate.
Not only does it look correct, it looks correct with an extremely Subject Matter Expert degree of authority. Often I'll work with an LLM, and it simply just misses so many things. I've worked in all sorts of different domains, software, chemistry, material design, everything from power generation through to physics, and in each and every case I see it missing incredibly important things. Any true subject matter expert would immediately bring up and prompt concerns, but not the LLM.
This makes sense, of course, because these are language models. They were trained on language. Their first and foremost capability is language.
An LLM's true expertise, true subject matter expertness is language.
And so anyone working with LLMs who isn't already highly skilled in the field they're asking questions about, will invariably be led astray and miss extremely important parts of a puzzle that need to be solved.