logoalt Hacker News

danpalmer • today at 3:50 AM • 0 replies • view on HN

Assuming the LLMs did actually run the tests, I think this points to the flaw in TDD and LLMs following goals blindly.

Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.

The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.