Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.

Only if we require the knowledge to be built into the weights. Give it access to a search engine and a big library of ebooks, and it might do better.

Doesn't matter if you aren't asking the type of questions where hallucinations are relevant e.g. you're seeking pure reasoning rather than factual information.

Probably. You can solve it with either some grounding context, or spending hundreds or thousands a month extra on a model that has more knowledge baked in. With modern harnesses, the choices is obvious.