This article feels like the result of having unlimited tokens at your disposal. It’s got some very expensive practices, where near-free practices would produce the same or better results.
There’s a long section in the middle where the article describes per-change verification via skill files and “ask Claude to use the chrome-for-Claude tool” when checking if changes have introduced regressions.
But there’s already tools which do this without spending tokens tokens. Playwright will achieve equal quality but deterministic results across the same verification attempts. It is write-once-use-often, so has a marginal cost near zero.
It feels like that because it is marketing to improve distant-customer results while also increasing their token use to grow revenue.
> It’s got some very expensive practices, where near-free practices would produce the same or better results.
I think tokens are near-free already, even for those of us outside Anthropic.
More importantly, your philosophy comes from a time where thinking had to be front-loaded: you try to predict everything that can go wrong, then design deterministic tests that can catch those things and a decision tree to figure out what to do in each case. This approach was the only viable way until now, but maybe that will change. Having the equivalent of a senior engineer thoroughly review each error with full context of what the application and feature is trying to achieve could perhaps uncover different insights, or cover edge cases more smoothly.
> This article feels like the result of having unlimited tokens at your disposal.
Gong to be honest where do you think this is going?
Do you think tokens are going to magically become more expensive or models are going to become less efficient?
Of course you should be tokenmaxxing, you can already tokenmax with a model that competes with the SOTA 1 year ago on a moderately powerful laptop you can walk out of an Apple store with.
I had a developer friend who wanted to automate gym reservations at night, his first idea was to make use of ChatGPT Computer use, but he encountered refusal of the AI to enter his password, and it was difficult to convince it to do it without approval as well.
I just told him to make the AI write a small selenium python script to do the clicking and it worked amazingly.
What makes you think they are not using Playwright ?
For me this article is more aimed at devs that are still stuck treating code generation and verification as two separate tasks. Giving agents the right tools (including playwright) to do both at once makes things more token efficient, not less.
Full app testing by agents would mean that each agent needs its own running app and browser use. So machines would cost too, not just the tokens in my opinion.
I got this feeling a couple of times this year with their sharing of best practices. They seemed theoretically interesting, but not if you're cost-conscious. It seemed like thinking with an unimited budget. And dont get me wrong, maybe that's the way to go since you assume costs of compute will fall down eventually (and these systems will become more optimized), but it would be completely untenable to run if you don't have a similar budget constraint (or lack thereof).
They are living in the future.
I think future AI cost will be as cheap as doing an API call in classic sense.
Yes. Anecdotally, I saved a ton of tokens (and bigger: time!) switching from having Claude drive Android emulator VMs (even with detailed skills and helpers) to using Roboelectric tests.
Exactly.
If you're doing this, the agents should create deterministic tests from its process rather than Agent tests or just writes deterministic tests for start. This is a bit of an absurd approach.
The issue I find with these articles is that they are basically the same boat as "Facebook Engineering" articles were 20 years ago.
People will be "oh because Anthropic does it it must be the way", then a bunch of influencers will retweet it as "anthropic kills writing tests", which some CEO's and team leads will parrot and get their team to absorb these practices even tho they are not the best practices.
Some of these things are just reinventing the wheel, like this is a basic script called "run_simulator.sh" and a pre-commit test hook, not a skill and "commit gate" that needs to check session transcript:
>To codify how to start the simulator or run your app, create a skill called verify. Put it in a .claude/skills/ folder in your repo, as close to the relevant code as you can, and add as many as you need. Claude knows to use the verify skill before committing changes.
>We've also created a commit gate that can block a commit if the session did not contain /code-review or /verify. We've set it up with off, observe, and enforce modes.
Wait, are you saying a company that makes money from selling tokens generated a content piece recommending people solve their problems by using a lot of tokens?
Scandalous.
Here's how to properly use XYZ, yes it'll cost you more but we manufacture it so to us it's no issue.
Do you not see the propaganda through the pseudo helpfulness?
e2e tests (which you describe at the end of your post) are literally the first thing the article talks about though.
> This article feels like the result of having unlimited tokens at your disposal
I think it's worth designing around that because tokens will only get cheaper over time.
Chinese gonna beat anthropic because they gluttonous use of AI.
Its almost become a meme at this point: well actually you can achieve the same using X so why waste tokens?!
The reason you should use Claude and not playwright always is because the the invariant you wanna capture is at the natural language level.
Like imagine I wanted to verify that Claude Code works and doesn't break. I can describe the invariant I want in natural language. Like: make sure that subagents don't block the original thread.
Doing this in playwright means that you have already bought into some previously existing architecture which means your tests are now correlated to your code. Having the invariants written in natural language allows you to decouple code vs the invariant.
I expect a lot of greybeard conservative programmers to keep opposing using AI here until eventually, not using AI will be seen as more risky.
You don't get a nice juicy IPO valuation by trying to save money or be efficient with your token usage!
It behooves the token merchant to espouse the benefits of having more tokens.
Not that it is an explicit, conscious direction, but it very much is one.
You’re not wrong, in that free flowing tokens are enough to convince such a group it’s simply good advice.
I agree completely, there was a much better talk on this by Lauren Tan: https://www.youtube.com/watch?v=O_zcc6Xl12g
A key pillar was introducing things like static analysis to provide cheap deterministic checks as the first goal and only then moving to other forms of verification.