alt Hacker NewsI have found it extremely helpful to keep the “token prediction machine” metaphor front and center in my work with llms, not because it expresses some fundamental limit of the technology but because it allows me to have a mental model about where and how to use the technology in my process.
Contrast this to the “chat” mental model. If you are chatting with someone and they lie to you, that is a transgression. A bad prediction is just a normal outcome. These are _human_ judgements but you address transgressions differently than normal but incorrect outcomes. That’s a helpful distinction for me when building my tooling.
If anything, that we are getting so much traction out of text prediction is the most magical part of llms. But it doesn’t change that you shouldn’t treat what they output the same as what a human says.
Hmm, what specific differences does it make in practice?
For me, the huge one is that LLMs are currently bad at learning from experience. I don’t trust any kind of automatic MEMORY.md or whatnot; in fact I greatly prefer starting from a clean slate each time because the LLM’s baseline general knowledge is so good.
In terms of accuracy and “lying”, I don’t really see a huge difference. Most LLMs are unfortunately a bit sycophantic and over-confident, but you sometimes see that in people as well.