alt Hacker NewsIt is completely possible to ask an LLM a series of increasingly more esoteric and discrete questions until you find precisely what information did, or did not make it into the model. If you know something rare and the LLM does not, you'll immediately see when it's hallucinating an answer or answering factually.
Replies
>>> I'm not anthropomorphizing anything ...
Yes you are, regarding LLMs at least. Here's why:
just for fun I asked a reasonably smart LLM to ...
[be] capable of understanding if it's gone off on
a hallucinatory path ...
You then went on to state:
> If you know something rare and the LLM does not, you'll immediately see when it's hallucinating an answer or answering factually.
Again, "hallucinating" is not something an algorithm can do. Also, determining factuality is again subjective based on the person assessing the information.
My favorite test is to just ask it to add two very large numbers or do other math of that sort. (this is also part of my favorite answer to the chinese room).
You'd be surprised how few digits you need to make a problem that is presumably unique in earth history. For a typical sum, the number of pre-existing answers would need to scale with 10^n lines of text where n is the number of digits. This expands out of control REALLY quickly. A quick guesstimate has you somehow reading out of a literal black hole at n=21 digits if your LUT is on paper, or n=26 digits if you're using modern HDD technology. O:-)
Now this is true, I do agree with this. There is indeed a good amount of memorization that is still taking place; see [1]. But it definitely isn't all memorization, or indeed, majority memorization. And even if we are able to extract things verbatim like this, we do not know how this is stored internally, as this may simply be the text that, with the rest of the internet in context, can be very radically compressed.
But in general, yes, the LLM cannot know about concepts that are far outside of its training set. Humans are the same, I would argue. If you add a good amount of your own knowledge into its context, or better yet, into finetuning, you might find it surprisingly easy to get it caught up on that material.
[1] Ahmed, A., Cooper, A. F., Koyejo, S., & Liang, P. (2026). Extracting books from production language models. arXiv preprint arXiv:2601.02671. https://arxiv.org/abs/2601.02671.