alt Hacker NewsAI is way beyond conventional LLM architecture now. It combines LLMs with search + RL. The traditional LLM architecture hit a wall around GPT-4o. Arc AGI evals show this.
AI is way beyond conventional LLM architecture now. It combines LLMs with search + RL. The traditional LLM architecture hit a wall around GPT-4o. Arc AGI evals show this.
All that extra is clear as day compared to the mystery of how neural network training decides to divide and balance the weights in even small neutral networks.
We can, at best, approach a good set of weights, even in tiny neural networks.
Imagine if we found a way to calculate the exact optimal weights for a given loss function. I mean, there is an exact optimal solution, it exists, but we can't find it exactly, even for a neural network with just 50 parameters.