Just think about what "tests pass" means for a rewrite. If you rewrite from language A to language B then any unit tests have to be rewritten during the rewrite.

So either "tests pass" does not include unit tests or unit tests were rewritten probably by the same AI that is doing the rewrite!

People are forgetting that "if the tests pass" is not a good measure of reliability. CS 101 from Dijkstra: tests cannot prove the absence of bugs.

Almost no test suite is comprehensive enough to cover even the kinds of bugs that will appear commonly in wide production use, let alone all of them. A big rewrite needs a lot of actual use as well to shake out the issues not covered by the test suite.

Given that even the unsafe audit article appears to be written by AI that doesn't seem like much thought to me.

The ones who accept rewrites are expected to support it indefinitely.

That's why Jared didn't consider using it for Bun even if it's undeniably better, they don't have human resources to support it

> quite a lot of analysis went into the rewrite

Literally just prompted for an LLM to review it and asked for a fancy presentation. That is not "quite a lot of analysis". That is anything but.

> If the tests pass, then why not accept the rewrite?

Because (1) tests passing are absolutely not a guarantee that no regressions were introduced in a change, and (2) even if they were, those tests are the result of thousands of hours of human labour, which is all well and good for the codebase as it currently exists, but who is going to be writing the tests for the 1m loc repo of unread code in the future? Unless you've proven that specifically LLM-generated tests can prevent all possible regressions, you're condemning the future of the project because nobody will be able to continue writing robust tests.

We've repeatedly seen that these test-driven LLM rewrites consistently produce absolute garbage.

[dead]