alt Hacker NewsWhat ends up happening is that your most fundamental features end up rotting because manual testing has biases. Chief among them is probably Recency Bias. It is in fact super easy to break a launch feature if it’s not gating any of the features you’re working on now. If you don’t automate those, yes, you’re nuts.
One of the worst ones I ever encountered was learning that someone broke the entire help system three months prior, and nobody noticed. Because developers don’t use the help system. I convinced a team of very skeptical people that E2E testing the help docs was a higher priority than automating testing of the authentication because every developer used that eight times a day or more. In fact on a previous project with trunk based builds, both times I broke login someone came to tell me so before the build finished.
Debugging is about doing cheap tests first to prune the problem space, and slower tests until you find the culprit. Testing often forgets that and will run expensive tests before fast ones. Particularly in the ice cream cone.
In short, if you declare an epic done with zero automation, you’re a fucking idiot.
I think maybe - this conversation is more about giving some more acknowledgement to the other side of this issue.
It's not that I disagree with you essentially - or particularly with respect to your analysis of your specific examples. 100% in the cases you describe. Those sound like beneficial tests. Particularly because your example SPEAKS to the business case - users were using the help docs (I think you mean users anyway). So yeah - that's important.
But I don't know why it's so hard extracting a simple acknowledgement of what I'm pointing out - specifically that the decisions like implementing tests IS a cost-benefit decision dependent on business context.
Funny you mention auth testing though. One time both me and the tech lead broke one of the auth flows in production within the space of a week of one another. Yep - no tests. Feel free to judge us insane. But here's how we thought about it - and when I say "we" that includes the business. First of all the auth flow was not actually used by any active users, so damage was low. Two man dev team. Complexity up until that point had been low, pre-product market fit, sales were dogshit, and cash had been low for some time. Feature shipping was the 110% priority. Ok - but these bugs were a sign complexity had increased beyond what we could manage without some tests. And given the importance of auth, it was now easy to make the case to leadership that implementing an e2e test suite was worth it. So we did.
If you still think a decision making process like that is insane - because we didn't immediately implement tests for every shipped feature. Well - I just think you're wrong.