alt Hacker News> Let’s say you have an incident, and you need a resolution. Do you send out the AI for the hunt and fix it within 5 minutes, or spend 30 minutes going around, figuring out the problem yourself?
It is far more often the case that you will spend more time getting the answer out of the LLM than you would've gotten yourself. They suck ass at troubleshooting systems, very frequently jumping to completely implausible scenarios that no human would think of as the first thing to try, and almost never asking questions to clarify the situation the way a human does.
Replies
> They suck ass at troubleshooting systems
Given the right tools and instructions I find them to be consistently superhuman. It's unreal.
I have an automated system responding to issues occurring on oceanconnect.ca, and every single diagnosis has been correct for months. Of course I still verify, I tune solutions, I review everything. But it's right. It's not a trivial system. It has a fairly large frontend, a large-scale model processing pipeline on the backend, a fairly complex database, and a queue for getting new data into the system. Claude does a stellar job of navigating and understanding problems, even when they're not 'real' or as they initially appear to be; things like quirks in old versions of Safari, or the model pipeline fails due to data integrity issues from the providers.
I've rarely encountered cases where I need to steer a frontier LLM much when troubleshooting. They're exceptional.
Sorry, this might be a thing in your org, and I have no idea hoe I can convince you, but it’s just not true in our environments. Maybe you might have a very non-standard set up, but frontiers do amazing job.
This has not at all been the case for me. LLMs (even the more-capable local models) are exceedingly good at troubleshooting when given proper context (clues, hints, a decent system prompt/skill that provides even the barest hint of how things work)
What you're describing sounds more like what would happen if you gave a chatbot without access to the repo as little context as possible and asked it to come up with a solution, blind.