logoalt Hacker News

spacedcowboy • today at 10:39 AM • 5 replies • view on HN

> I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI"

Me too, my first neural network research was for my PhD, 35 years or so ago, on a Meiko computing surface.

My bar for AGI over what we have (generative AI or genAI) is still fairly high, fortunately I do not believe that the current genAI path is set to lead us to AGI, there are very fundamental differences.

Right now for genAI, the model is static, the weights do not change, all “memory” is in the context, and even with the context sometimes exhibiting “J-space” (a shared "workspace" for deliberate, multi-step reasoning spontaneously created in the context memory), its ephemeral; it doesn't inform the weights in the base model. Even when we try to do “weight-space programming” to actually modify the model itself, the supervisor-model needed to do that is far and away larger than the one being operated on, and once a change has been made the supervisor-model is now out of sync with its original representation and it becomes less useful the more it diverges. We have not solved "thinking" today, we “just” have a high-probability path between states in a static weight-space which we follow and feed back.

What thinking requires is the ability to generate hypotheses (which requires internal state modification), to hold them and test them against other facts and hypotheses, and to update the internal state on test results, negative or positive. LLM’s don’t do that, they’re actually more like a database lookup, where the input tokens act as keys into a high-dimensional space where information is stored. You could even consider “the model” to be just an entropic compression of lots of data, and the inference step to be the lookup. It’s the scope, and the fuzziness of the lookup that makes them seem to work so well. Lots of results come back, and the best-path through those results wins. It’s not “thinking” as we understand it.

This is why humans are still a crucial component: we set the boundary conditions of these lookups, we set the starting point and the goals, and we adjust the success criteria to guide the chat - note that these are all the comprehension parts of the task. It’s also why models will hallucinate: the fundamental design does not lend itself to a null hypothesis. Even in the newest "reasoning" models, there is still no feedback of state back into the model itself. What they are effectively doing is running a Monte Carlo pass through the problem space—generating hidden branches and picking a likely path—before reporting back. Because the underlying weights never update from that process, it’s still just a sophisticated search, not genuine comprehension - there is still no “that makes no sense, reconfigure” step.

Stuart Kauffman (“The Origins Of Order”) would say its weights are stuck in the frozen realm of order, and that it takes a genuine phase transition to get to where the real action is - at the boundary of chaos and order where humans reside. Tweaking and fiddling with the system while it remains locked in the stasis of static weights is never going to effect that state change.

That’s not to say genAI is not useful, it’s freaking awesome, possibly the best invention in the history of time, but it doesn’t truly “think”, it maps.


Replies

lordnacho • today at 1:59 PM

What I find is that the LLM is like a smart person who knows when to use a tool. And it is freaking good at finding tools to use.

A bit like a person who is smart enough to figure out how an excavator works, or a drill, or sewing needle.

It doesn't feel like it ever internalizes anything, it just knows there's a million tools available, and the extent to which it thinks is that is can do enough to plan some tool uses, look at the result, and continue until some goal is reached.

When it comes to hypothesis testing, that seems to be what it does. It says "hmm, I'm going to open up my hypothesis testing kit (typically python) and drive this data through it". At least this is what I've seen when I put some statistics through it.

In general, it seems like the intelligence is mainly a kind of glue. It knows enough about what "right" looks like in each domain, but as a sort of consumer. He doesn't bake the cake, he buys it in.

kenjackson • today at 12:37 PM

It seems like your definition of AGI is too focused on implementation details. IMO it has to be focused on behavior. Once you say something like “the weights do not change” then I think we’ve lost the plot. Instead say this is what an intelligent being can do, but a GenAI cannot.

➕ show 2 replies
mark_l_watson • today at 4:39 PM

+1 for the Stuart Kauffman mention. I was in Sante Fe many years ago and looked him up, one of the enjoyable talks I have had in my life.

I really dislike so much effort put into LLMs. We really need to widen approaches for AI.

Lambdanaut • today at 3:46 PM

This was beautifully stated. I really appreciate you taking the time to write this.

curt15 • today at 11:44 AM

Self-modifying neural nets?