alt Hacker NewsI disagree with the assertion that the model gets it. Here’s a practical example I just tried with Fable 5.1. I gave it this prompt: “Write a Go function that can be used to establish secure communication to a remote system using a certificate. Keep it short, single function, and explain how to use it.” The output forced the use of a private key stored in a file even though that wasn’t specified anywhere as a requirement. The function Claude wrote takes a private key file argument and calls a Golang function that requires a private key file (tls.LoadX509KeyPair) even though Go has crypto.Signer which could support private keys in various other manifestations like HSM or KMS. I argue that a person who “gets it” (or who is reasonably experienced in security) would have opted for not requiring private key material for this to work.
For the record, this isn’t unique to Claude. ChatGPT and Gemini do the same, each with its own quirks. ChatGPT got extra credit for being the only one who allowed the function to also take a CA file for server authentication.
Don’t get me wrong: LLMs are the future (maybe even the present) of software development but I think there’s some way to go before they can be entirely hands-off in some areas. I still find myself having to course correct designs and plan mode helps me with that.
And of course, thank you for your work on Claude. :)
Replies
Whether or not a distinct "plan mode" is needed, upfront planning remains essential in my experience, even with Fable (albeit not the 5.1 version). I agree that, as the models get better, you can skip planning on increasingly complicated tasks.
But there is still a ceiling above which it is necessary to "preload" the context window before starting to call tools and get into the meat of the work. You want to establish domain language (especially with Claude models which otherwise will invent their own, and it will be inscrutable) and key requirements and assumptions. You want to do a Q&A iteration cycle with the LLM. You definitely should do a sanity check that the LLM actually "understands" what you were trying to achieve, and then make sure that understanding is coherently and plainly stated in the prompt. All of that seems to be necessary still for just about any serious task, if you actually care about the quality of the results and/or don't want to burn hundreds of thousands of tokens on flailing around to get to a good quality result.
So no, you don't "need" plan mode. But you do still need to do all of the things you would do with plan mode.
But in your example you never even asked it to plan so you could check the implementation before writing it you just asked it to write it directly, so this isn't even a comparison to plan mode
This is a pretty obscure and in-the-weeds benchmark, but to me the models’ interpretation feels quite reasonable.
It can't read your mind that you have an HSM, can it?
[dead]
I'm hoping that by "gets it," he meant that if you start a discussion about the design, it doesn't misunderstand and immediately go off to do the work. Some models tend to do this.