logoalt Hacker News

resters • today at 2:52 AM • 3 replies • view on HN

"one-shotting" is fun for a low (human) effort toy, but it's a mistake to get overly hung up on LLMs getting everything right on the first try, since a lot of that is the harness and the internal prompting that makes it burn extra tokens to test everything to get it in that one "shot" (ui turn).

While it's nice that LLMs are getting better at arriving at the desired deliverable with fewer surprises (or broken things) doing it is not necessarily a sign that the model is actually smarter, just an indication that what you are doing is likely a well-trodden path for the model, the RL and the harness.


Replies

aenis • today at 3:08 AM

But for how long?

I have a personal benchmark suite that runs a bunch of difficult algorithmic problems in 3 tiers: vibe coded prompt (just a few sentences), vibe coded prompt plus a description on what the solution will be evaluated on, and finally a fully specc'd prompt that leaves little space for interpretation.

The current Fable models are approaching a level where the -eval runs perform systemicallty better than the "clear spec" runs. (And right now its just Fable 5.1 and Opus 5.5 that behave so well, all the other models I tested - all the OpenAI and the leading Chinese models - work better with clear specs). I'd encourage everyone to experiment a bit; I am running my serious workflows with labourious force-me-through-all-the-design-questions mode, whereas my hobby projects, some of them hard, I am trying to see how much I can get away with with vibe-grade prompts. My conclusion is, with Fable its starting to look like what Vibe coding was promissed to be a year ago - a magical box that generally does things very well.

Btw, congrats to the OP -- the game is enjoyable and I played a few levels. I dont remember when I last played any games for more than 5 minutes. Good stuff.

➕ show 5 replies
tty456 • today at 3:23 AM

But one shot prompting does not mean one shot testing. I see continually that Fable can design and test prototypes before committing to a design goal and testing multiple layers until the goal is reached. Many of the tests are code unit or integration tests that can be re-used without token spend.

locknitpicker • today at 6:16 AM

> "one-shotting" is fun for a low (human) effort toy, but it's a mistake to get overly hung up on LLMs getting everything right on the first try, since a lot of that is the harness and the internal prompting that makes it burn extra tokens to test everything to get it in that one "shot" (ui turn).

I'd be more specific and state that these models are now way far from being LLMs and instead they are full blown agent orchestration systems capable of running multi-layer ReAct loops and tool-calling, and with your whole session history as context. It doesn't make any sense to refer to them as glorified chatbots as the LLM part is nowadays far from being the component that delivers the highest value.

➕ show 2 replies