An interesting talk I heard at a conference once, that I can neither remember the speaker for or speak to their legitimacy, suggested that we might have some lower form of intelligence encoded into our language. They posed the idea that we have enough unique words, and combination of words, that it starts to have reason unto itself similar to how our neurons and their connection breed intelligence. The idea was that we as humans have baked intelligence into our own speech patterns. It seemed a little to abstract for me, but potentially goes a little way to explaining how a statistical averaging algorithm with some randomness, at scale, starts to look like it very occasionally has a genuinely novel thought.

The cure for HER2- metastatic breast cancer is a simple matter of ...

Please predict the next word.

Intelligence is implicit in language understanding. The best possible next-word-predictor is omniscient.

Nobody knows how it works, really. It just turned out that if you try to predict the next word then you get intelligent behavior, depending on amount of training data, and the size and topology of the network. But again, nobody knows why, and what the limits are.

Here's my grok of it: Deep learning models progressively abstract a concept presented at the input by passing the input through many sequential layers () until an output layer transforms the output of the final layer into something interpretable, such as an indication of what token to predict next, or a classification, or whatever. The transformer architecture futhermore offers layers that allow different parts of the previous layer's output to sort of mix with each other in complex ways. As you get into greater levels of abstraction, the attention process is mixing very abstract concepts with each other in a nonetheless highly structured manner. I believe this is where the intelligence lives.

sometimes with residual connections, but we can ignore that for sake of simplicity.

Intelligence as a measure of the ability to define predictive models of certain problems (and their solutions).

Promoting LLMs is encoding the problem we want into the query vectors, and through the magic of the complex training and the power of operations in a very large dimensional abstract space the AI can manipulate the representations, and iteratively approximate solutions. (And using bigger and bigger contexts and better encodings it can form better models.)

Biochemistry is abstract to us humans too, we can only create hypotheses, and validate them experimentally.

Compression and understanding are correlated.

I don't think LLMs currently have direct reasoning abilities, but as we make them more complicated (MoE, RL) I think we're getting better at learning an implicit world model that guides the token output distribution towards making good hypotheses.

How do you think? I think with words.

AI is way beyond conventional LLM architecture now. It combines LLMs with search + RL. The traditional LLM architecture hit a wall around GPT-4o. Arc AGI evals show this.

Granted, I feel like munging gigabytes of text data (i.e. G, A, T and Cs) would be something LLMs would be good at

I imagine it's writing a story about a character doing those things and then reading the story and acting on it.

I don't see why its that crazy that a system with a huge amount of parameters starts to exhibit emergent behavior

No worries, no one does. Exactly like no one knows how the brain reasons either.

I don't think anyone knows, not even the LLMs.

I mean, the subtlety of the neural network weights that emerge from training are not fully comprehended by anyone, man or machine.

Every individual calculation is understood, and every step of training is understood, but the exact nature of those weights that divide the responsibility of responding to subtle changes of input in intelligent ways is beyond me.

It's really good at pattern recognition.

So I'm not sure how it knows to be 'surprised' that alone is pretty fascinating.

[dead]