alt Hacker NewsIt’s also a great way for AI to keep creating backwards compatible bloat code when refactoring and whatnot because “tests are failing” when those tests are testing absolutely silly stuff (LLMs have a tendency to want to validate exact output rather than behavior).
I don’t think anyone has figured out the proper solution yet, I personally try to get the LLM to write code similar as I would have written it myself.
Replies
What do you mean "exact output rather than behavior"?
If you mean asserting against hardcoded literals, then that is the preferred testing shape in most cases. And for good reason.
If you change production code and a unit test fails, that is useful. Idk why that's silly. If you want to test more complex interaction write a UITest or integration test.
I honestly think we're all living in different worlds. There's probably large differences in capability on a per-language, per-team, per-codebase, per-engineer environment that are producing wildly different perceptions in output quality.
LLM's are way better at generating realistic tests than any engineer I've ever worked with as long as you are pedantic about asking for it and making sure it follows it.