A TLDR, since most comments here seem to misunderstand the result.

The idea is: a model may only have 100M neurons or whatever. But there are infinite concepts in the world. Therefore, models have to "stuff" multiple concepts per neuron.

How do you do that? Theres two ways:

1. Each neuron handles similar concepts. One neuron for dogs, a separate neuron one for cars, etc.

2. Each neuron handles a bundle of unrelated concepts. One neuron handles dogs and cars. Another neuron handles cats and numbers. And so on.

Historically, people thought #2 was correct because many neurons were found that seemingly represented multiple concepts. #2 had a cool name (polysemanticity) and was promoted in OpenAI/Anthropic blogs so it had a lot of traction.

However, this blog says #2 is wrong. Actually, the dog/car neuron is just a "V shape neuron". So #2 was just methodological error, and #1 was right all along. This is all the blog is saying.