alt Hacker NewsI think it has more to do with LLM's being statistical models than human creativity lacking in the input. The creativity and millions of voices and tones may be there, but since these models tend to go for the most likely next words, polishing this away becomes a feature.
A text by a human mind may be seen as a jagged crystal with rough edges and character. Maybe not perfectly written but it's special.
An LLM takes a million of crystals and trims the most likely tokens to be chosen into what would rather appear as a smooth pebble; the common core of all crystals. And everyone using the LLM will get very similar pebbles because to the LLM, regardless who is speaking to it, it will provide the same most likely next tokens. It's not that creativity is lacking in the input, but the LLM picks the most commonly chosen words by all humans in given contexts.
For that to sound imaginative and great as you go, it would have to not only exist in the data, but be a common dominating voice among humans. But if it was, it wouldn't be seen as creative because it would be the new normal.
So I'm not sure how there's a good way out of this. You could push LLM temperature high so that it becomes more "creative" by picking less popular tokens as it writes, but this instead tend to make it unpredictable and picking words it shouldn't have. I mean, we are still dealing with statistical models here rather than brains and it's a rough tool for that job.
>I think it has more to do with LLM's being statistical models than human creativity lacking in the input. The creativity and millions of voices and tones may be there, but since these models tend to go for the most likely next words, polishing this away becomes a feature.
I have always thought this is a rather misguided view as to what LLMs do and indeed what statistical models are. When people describe something as 'just statistics' I feel like they have a rather high-school-ish view of what statistics represents and are transferring this simplistic view to what is going on inside a LLM. Notably they do not find the most probable next word. They find the probability of every word that could come next. That is a far richer signal than most imagine.
And ultimately it's like saying that human brains are just chemical bonds changing and sometimes triggering electrical pulses that causes some more chemicals to change. Complex arrangements of simple mechanisms can produce human thought. Pointing at any simple internal mechanism of an entity without taking into account the structural complexity would force you to assume that both AI and Humans are incapable of creativity.
Transformers are essentially multi-layer perceptron with a mechanism attached to transfer information to where it is needed.