GP did not suggest anything of the kind, though. They talked about "figuring out where to look" which very different. In that kind of use, it doesn't really matter that the model might hallucinate or draw faulty conclusions, as you're not using it figure out what to do, how or why. The only thing it is telling you is where some information is located, after which you go check it out from the actual source of truth. If the reference turns out to be wrong, you'll notice immediately.

Plus everything important is always written down.

It’s more true in aviation than anywhere else (except maybe space flight), but still.

I hate AI as much as anybody, but claiming that it's not useful for "distilling volumes of data into a short list of places to look first" is silly.