logoalt Hacker News

pianopatricktoday at 5:36 PM1 replyview on HN

At what point should I trust projects like this?

I've heard stories of LLMs changing tests to get all the tests to pass instead of actually fixing code. So I'm not sure I trust it just because tests pass. I also think that tests do not and cannot check everything.

I also don't trust it just because it compiles in Rust. The Rust compiler does not check everything.

So at what point do projects like this cross from "untrusted" to "trusted"?


Replies

simonwtoday at 6:13 PM

> I've heard stories of LLMs changing tests to get all the tests to pass instead of actually fixing code.

That's very easy to prevent. Don't let them edit the tests! Run the test suite against a reserved copy.

show 1 reply