alt Hacker NewsIs anyone here working on a problem for which current generation LLMs are inadequate, but that could possibly be solved by the next release of a first tier LLM?
Or is it like bicycles? Unless your problem is named Tadej, you don't need a $13,000 bike.
Replies
For me personally, the answer is no. Fable is adequate to do basically anything I want to do. My perspective, broadly speaking, is that we've saturated most of the benchmarks because we've largely saturated our capacity to verify models' work at scale. What's left is context-bound verification, i.e. the problem of ensuring that output matches intent and ambiguities in prompting were resolved correctly. Further advances in autonomy do not make that latter verification problem easier. If anything they make it harder as the output per task becomes more complex and therefore more taxing for a human to verify.
The solution to that (to my mind) would be not a better model but a basic shift in architecture beyond the current paradigm and into a setup where agents have durable, plastic memories and undergo contextual individuation over time. But at that point agents start to become quasi-persons and not tools.
I asked a current generation LLM to make me $1k a week and it hasn't so far.
I want to be able to generate my own Simlilirian movie by dumping the content of a book into an LLM.
Both animated and live action results would be acceptable.
Unfortunately most existing LLMs lack the capability to maintain context across tens of thousands of frames.
I saw a laptop earlier in the train that I asked ChatGPT, Claude and Gemini what it was, providing a brand, screen size and ports description. Gemini could never figure it out, Claude and ChatGPT eventually did, after multiple rounds of indirection, giving completely wrong answers (there was a perfect match for the problem statement, they all explored alternatives first). LLMs are (probably) amazing at things I don't care about, and still suck at the mundane stuff you would have the marketing tell you they excel at.
3d modeling to an STL a part compatible to a visible cable raceway still fails even if I let Claude Fable use me as a robot that measures with calipers.
I was given a picture cube, which is like a Rubik's cube but every side is a unique picture. It came scrambled and I don't have an original reference image. I like to take videos of it and give it to llms to solve. I call it my agi test because it hasn't been solved yet
Yes. Most of us are, still. The frontier is currently both at expanding ‘common sense’ / non-cheating outcomes for imprecisely specified software (that’s all software), and at expanding autonomy - ability to work longer unsupervised with success, oh, and also at expanding outside contextual reasoning about what’s being built, as in “hmm, that doesn’t look right or make sense, let me explore that.”
Scientific physics simulations - even the frontier models just engage in rationalization of obviously unphysical results instead of understanding the system. They have the rote knowledge but fail to apply it unless their hand is held through the process.
I think the tech analogy for frontier models is going to be super computers.
Super computers keep getting better but most people don't need them for most things.
Hardware debugging and firmware details lead to thinking/testing loops on all but the frontier here.
Yes, lots – I think that folks will hopefully discover more of these as they scale up their ambition, now that LLMs make a lot of previously difficult things far easier.
This is the exact same type of comment I heard about computer hardware upgrades for three decades in a row.
“Very few people actually require a Pentium workstation, a 486 is perfectly adequate for the majority”
The logical fallacy is taking an extant distribution of “product capability” that is priced to fit what the market will bear and assuming the “next upgrade” simply tacks on a little bit more to the right hand rail of that curve.
No!
It shifts the entire curve!
Everything for everyone gets better and the top 1% of the most demanding users will continue to pay the same-ish premium.
“Nothing” will change.
Look at it this way: you can buy a $200 laptop for your kid or a $20,000 Mac with an M5 Ultra processor.
BOTH are vastly more powerful than either a $200 PC or a $20,000 “workstation” from 20+ years ago.
Look at: https://arena.ai/leaderboard/text?q=openai&utm_source=chatgp...
The “budget” 5.5 Instant model beats o1 and o3 which were “pro” models at the time of their release!
Infra as code and devops shit. Fable is there in general because things it doesn’t know I can point at documentation and have it do a reasonable job. Opus 5 sucks. If I don’t have fable quota, I drop to Opus 4.8 and hold its hand.
[dead]
I get buy with very cheap models and actually using my brain, you don't need these SOTA models. China will definitely win this AI 'war'
My experience is that even Opus 5 still tends to write buggy or low-quality code and makes serious mistakes when analyzing code. It's a lot better than before but still not something I trust. I've had less experience with Fable since I can't use it at work; I hear it's a step up but still has its limits.
For large tasks like a web browser or a compiler, even expensive swarms of frontier LLMs have not been shown capable of producing codebases that actually work. (Anthropic built a C compiler with Opus 4.6 but it lacked optimizations and apparently hit a complexity wall.)
I also want to use LLMs for reverse engineering, but apparently it's pretty hit-or-miss, especially if you're forced to use open-source models to avoid restrictions.