alt Hacker NewsI've found that keeping very healthy test coverage solves issues like this. LLMs are very good at validating their own implementations with TDD.
Replies
With unit tests you gotta be careful though, oftentimes LLMs skip implementations with mockups that just say "not implemented yet" or similar and then the unit tests become pointless because they start to only test internal structures for being set / not default values.
For me it helped a lot to try to make containerized end-to-end tests and a custom TestMain for this, where I am using podman to run the integration tests. This way the end-to-end tests are forced to be on network level, and you can test protocol and API quirks much easier with LLMs.
Also, never forget to write a bootstrapping docs/ folder so that you don't have to re-explain these things all the time.
i think they validate less with TDD; if you tell them a bare bones spec to validate they tend to write just that. if you write the test after, then their testing is causally conditioned on what was written. if you are going to write tests, it seems like teat first is way better than test last.