logoalt Hacker News

aronaxe • yesterday at 8:30 PM • 1 reply • view on HN

Brain waves are to biological neural populations what large-scale, low-dimensional activation modes in the transformer’s residual stream are to an LLM.


Replies

tete • today at 12:00 PM

No they are not. Stop comparing LLMs with brains. Even way before LLMs and the AI craze people realized that ANN (Artificial Neural Networks) are a bad bane and bad abstraction. At this point they are at best "inspired by", but partly also based on misunderstandings.

Basically every new learning about how neurons and the brain works is increasing the distance between how neurons and "artificial neurons" work. There are so many ways in which they are different that I have to stop myself from giving examples, so basically google for it (or use an LLM if you must). The whole memory and computation part, the fact that there are many different types of neurons doing different things, the fact that so much stuff isn't still understood.

And it's not like ANNs are per se changed or adapted to come any closer. A brain or neuron simulation is something very distinct from an artificial neuron.

Yes, it's good enough to emulate something learning, it's probably even an excellent choice to build an AGI and despite LLMs being far from it I think that ANNs will be the right choice, simply because they are a simple basic piece, but they are still not what a biological neuron is. It's simply a different thing.

Also just to be clear: That doesn't mean to say that an artificial biological neuron cannot be made or that we will not be using that or anything like that or that neurons are somehow magical and cannot possibly be replicated or anything like that.

It's just LLMs, ANNs and biological neurons, brains etc. are widely different. Just like a stick figure, a doll or a (current day) robot, an NPC, a chatbot, etc. might be a usable "abstraction" of a human, but is very dissimilar nevertheless. Making artificial and biological neurons equivalent is like pretending your phone has feelings and a bad day, maybe needs to rest, because some app malfunctions. Maybe letting it sit for a while will make things work again, but it's not because it overcame its bad day.