> It would be weird, wouldn’t it?

That depends how it tried and what influences the LLM had on the outside world.

For example if the LLM could drug you, either by convincing you to take drugs, or trick you into doing it, then convincing you that you're not real becomes monumentally easy. Derealization can be a monumentally profound/terrifying experience to your psyche.