logoalt Hacker News

How the Claude Code team thinks about verification

185 points • by craigmart • today at 10:20 AM • 155 comments • view on HN

Comments

OtherShrezzing • today at 11:35 AM

This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.

There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.

But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.

➕ show 19 replies
tshaddox • today at 3:49 PM

Is Claude Code considered to be a reliable piece of software these days?

I stopped using as my daily driver around 9 months ago, and at that time it felt very heavy/janky and seemed to have very regular bugs and breaking UI changes.

I’d be happy to hear that it has gotten better!

➕ show 5 replies
girvo • today at 11:01 AM

> With them, you can run dozens or hundreds in parallel.

Im already spending my $2k usd per month in tokens with just one or two agents. How do they expect me to do that?

➕ show 6 replies
asmvolatile • today at 3:30 PM

The Claude code team doesn’t think about jack sh*t. Interacting with Anthropic teams makes it clear that nobody at that company has any idea how their software works.

➕ show 1 reply
Narann • today at 11:53 AM

The more I read articles like this, the more I realize Claude is good for process prototyping, but you need to "bake" the process with flat code/script/organization to decrease costs.

The future of IA would actually be to large scale prototype easily, even more like Python was a prototype language.

koolba • today at 11:53 AM

> Nobody can read every change, so we use failures as one way to decide what needs a person.

Whether a human can or should review every diff is a conscious project level decision. I have yet to see a project where it is not feasible. The trick is to break things into small enough pieces that you can review them. Just like one would do if you were doing such changes manually. Nobody in their right mind performs non-mechanical changes of 100s of files.

➕ show 3 replies
r0m4n0 • today at 4:55 PM

The rate at which software engineering has changed is staggering. We don’t fully understand what is the right or wrong way to be doing things. It’s cool that they share what they are doing but that’s probably about the only thing most people should take from this. Jury is still out on whether this is sustainable budget wise, can be done better differently, whether this sustainable maintenance or quality wise, etc

FartyMcFarter • today at 11:56 AM

> Flaky tests get the same treatment. Claude can't skip them, add sleeps or retry until they pass. If it can't find the root cause, it deletes the test and writes down why.

If the tests are hermetic, you should always be able to find the root cause (by bisecting to determine when the test became flaky).

If the tests are not hermetic, it should still be possible to identify the dependency that is making them flaky, and use that as a justification if it still makes sense to delete the test.

➕ show 1 reply
hliyan • today at 12:05 PM

In my opinion, the output of a stochastic process can only be truly verified through a deterministic process. Tests already do this for functional accuracy. Linters can do this for line-level code, with code treated as text. But there is currently no such method of deterministic verification for the structure of the entire program, or more importantly the viability of the architecture when you throw in connectivity and IOPS.

Perhaps we need a level above linters: a tool that constructs the abstract syntax tree and verifies its various branches against deterministic rules.

How to do this for architecture: unsure.

➕ show 2 replies
gregwebs • today at 12:41 PM

After I created this skill for verification, it completely changed my level of trust with AI coding.

https://github.com/gregwebs/skills-sdlc/blob/main/skills/ver...

The implementation process that I am using also looks for verifications during planning and reviewing and makes sure they are completed. It does use a lot more time and AI spend than just asking AI to do something. I still do the latter for simple or noncritical changes. I had to switch to using DeepSeek as my workhorse to keep down my spend.

https://github.com/gregwebs/skills-sdlc/blob/main/skills/imp...

nelox • today at 1:00 PM

We are well on the way to recursive self improvement

holoduke • today at 5:31 PM

What we need are ultra fast testing suites. Specially e2e tests are notoriously slow. Runtime is big and its often difficult to isolate just a few tests. I want 1000 e2e tests in a few seconds instead of 10 minutes. It's also a mayor blocker for an agent that cannot work fast because of the constant waiting. At the same time you need the test feedback loop in order for the LLM to do proper shotgun programming.

sashank_1509 • today at 3:42 PM

Take 10-20 skilled programmers, the kind that made web browsers, game engines etc. Give them a year. They will end up with a superior working version of Claude Code without thousands of bugs and issues, that keep getting fixed and unfixed and trillions of tokens spent by AI.

They won’t even do all this verification nonsense and their end product will be of superior quality.

anshumankmr • today at 5:10 PM

Honestly, the amount of bullshit unit tests Claude keeps pushing nowadays. I took a few hours the past few weeks, some with AI, some by myself to hand remove some really dumb fucking test cases. God knows how long the CI must be running when these guys push their code.

➕ show 1 reply
nandanadileep29 • today at 11:13 AM

excellent content from the team. very concise and clear

johnrc • today at 6:40 PM

[flagged]

tanmaycodex1 • today at 6:05 PM

[flagged]

arbor-group • today at 2:58 PM

[flagged]

adilbasari • today at 3:05 PM

[flagged]

fk202611 • today at 3:09 PM

[dead]

ankur_misra • today at 3:39 PM

[flagged]

oceanrelay • today at 11:06 AM

[flagged]

anatolii1309 • today at 3:22 PM

[flagged]

makercfd • today at 3:02 PM

[flagged]

dotdi • today at 11:20 AM

> https://github.com/anthropics/claude-code/issues has over 16k open issues

this u?

➕ show 1 reply
Ampersander • today at 11:05 AM

Why is the team thinking? Their product should be doing that for them

➕ show 2 replies
saagarjha • today at 1:19 PM

This is just a depressing read, and I’m not even sure why the team decided to post it because it just looks horrible. Like, you’re telling me that the team’s approach to verification is that they post Claudeslop blog posts talking about the most unimaginative ways to write tests? That their idea of making things work is this?

> You can also close the verification loop by using the results to improve the agent’s performance on that task. For example, if an agent repeatedly fails a test that checks logs for user data, you can create a hook or a more explicit rule in your CLAUDE.md to avoid future issues. This (mostly) stops issues from happening again.

There are people out there who are doing all sorts of crazy neat things in the verification space using the tools you build and yet the people who could be helping seem to have no good ideas on how to help :/

titzer • today at 1:20 PM

I see a lot about adding skills to make Claude test things.

How about making Claude generate scripts that you can run?

Algorithms are better than agents. In fact, the whole point of AI is to reduce thinking to algorithmics. I didn't see a lot of evidence that the team understands this. If you need a magic box to test your other magic box, you have magic boxes all the way down.

➕ show 1 reply
TheRoque • today at 12:52 PM

I can totally see that they don't review their product at all, Claude code is probably the thrashiest TUI I've ever used.

lavendermon • today at 2:11 PM

At what point in the future could the title become: "How the Claude Code team's Claude Code thinks about verification"?

camgunz • today at 4:24 PM

Congratulations you have invented testing.

inamorty • today at 11:40 AM

Can they not afford someone with an engineering background?

➕ show 1 reply
whamp • today at 2:16 PM

This whole thing read like google not anthropic.

matltc • today at 12:12 PM

Formal vibefication

jordemort • today at 2:13 PM

we think about verification

we don’t actually do any of the work but we think about it

ForHackernews • today at 11:25 AM

Ironic that this page loads partially styled with broken CSS:

"The resource at “https://claude.dev/_next/static/chunks/241gnzjfqbp9f.css” preloaded with link preload was not used within a few seconds. Make sure all attributes of the preload tag are set correctly."

dude250711 • today at 10:49 AM

> ...On the Claude Code team, the vast majority of our code (~90%) is written and reviewed by Claude...

I am more interested in the remaining 10%.

➕ show 1 reply
123ajhgF • today at 12:58 PM

Is the perfect verification the reason that Claude is down so often?

More snake oil and theatrical agent setups described in insufferable Claudish. I think the target group of Claude is people with an IQ below 100.

mococa • today at 11:34 AM

So a lot of slop...

i_rutul_patel • today at 2:31 PM

[flagged]

runtarmac • today at 11:40 AM

[flagged]

Yaojia666 • today at 12:15 PM

[flagged]

sick_of_slop • today at 1:14 PM

[dead]

yoy0721 • today at 12:01 PM

[flagged]

ahmedketata • today at 12:05 PM

[dead]

jonenst • today at 11:50 AM

[dead]

Eunice99x • today at 3:06 PM

[flagged]

mohsen1 • today at 11:20 AM

[flagged]

➕ show 1 reply