alt Hacker NewsIf the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?
Replies
1st line of defence, use something like ponytail to enforce brevity. Use property testing and behavioural testing on top of unit testing. Enforce readability standards so you will be able to understand the tests.
2nd line, code review.
Do the first pass with an agent, ask it to bounce back vacuous or tautological tests. Ask it to verify that the tests verify what they claim to. Then read them yourself.
3rd line, mutation testing. If the tests don't actually catch broken code, kill the mutants.
Read the tests.
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
Existing tests.
[dead]
Manual testing.
I work on user facing applications, and since the models do not have good taste, testing the UX is essential.
If you spot a bug, usually the model will attempt to reproduce it in a new test case that does cover the actual issue.