> not mostly the test, but the real world production scars

Most extensive test suites are exactly production scars: every time you have a bug or a regression, you write a test that confirms correct behaviour.

SQLite is a good example to bring up because its extensive closed-source tests are what’s often cited as being what keeps people from forking it. (Turso did it, though, but it takes a company to deliver some guarantee of equivalent diligence.)

And yes, years and years of running.

I agree. I also agree with the sibling reply that -

> every time you have a bug or a regression, you write a test that confirms correct behaviour.

What I fail to see in these rewrites however is - what about new bugs introduced by virtue of this rewrite? I mean it'll have to go through its own challenges in real-world scenarios, right?

> I start to see a lot of these re-writes that depend on tests to state that its working.

There's another way to validate the rewrite though. Just run both pgrust and postgres and compare the output. Know of an edge case? Run it too. Doesn't know? Use a fuzzer or some automated tool to find interesting inputs. Found an inconsistency? The input/output pair becomes a test case now

Not sure if there's tooling for that though. If there is, just give it to Claude so they will incorporate it in their development loop

Not weighing in on this specific rewrite but tests are how you specify that your software works correctly. If a behavior isn’t covered by an automated test in some form you can’t assert that any given change doesn’t break it.

I think it is completely reasonable to use a preexisting unmodified test suite to state that something is working. The larger the project the more true this becomes. Real world production scars are documented and guarded against in the test suite otherwise those lessons get lost.

Also SQLite is legendary for its massive test suite and extensive fuzzing. They have 590x the amount of test code and scripts than normal code. Source: https://sqlite.org/testing.html

Great point. Stated another way:

"Mom, can I have battle-tested, reliable software"

"We have battle-tested, reliable software at home"

Battle-tested, reliable software at home: (Pic of green text from `cargo test`)

> That's where the reliability comes from

So, we should make it easier to feed that reliability back upstream.

Probably the most useful thing you can do with these LLM-transpilations for now: If the transpiled version passes all original tests, I can run my application test suite against it and use it to discover test coverage deficiencies in the original!

If it crashes or otherwise observably misbehaves, I know the real project was missing regression tests for something. We could make upstream so much more resilient against accidentally breaking stuff in future updates, if only it becomes safe (offline + no side effects) and easy (if it crashes/locks, it is not from some memory safety bug from 25k transactions earlier) to run these transpiled projects as one row in our everyday integration matrix.

As sibling mentioned - bugs and regressions are the thing that are (in a perfect world) usually covered.

The problem however is non-covered success cases. A visualisation of the problem: let's say universe of interaction for DB consists of 10.000 SQL queries. Over 10 years various regressions were found and 2.000 SQL queries are guarded by tests. In reference implementation remaining 8.000 never surfaced over this time and it's unclear if they will work.

And, thinking of how many various SQL queries PostgreSQL users around the world are using vs the test cases covered it's obvious that feature space isn't covered in 1% of the success ratio cases.

Now the new, test-based implementation, has to prove it can handle remaining 99%.

Completely agree with this.

The biggest lie of software engineering is that everything can be testable with tests. That a 100% test coverage is an indicator of quality software.

And also the amount of people running it in thousands of scenarios. Not sure if these areas can be even tested for, but I guess time will tell (can observe Bun if it breaks somewhere as that’s afaik the first big AI rewrite which got into prod for masses).

That's precisely what a regression test suite is for. There is a bug, you fix the bug, you add a regression test. So if the test suite is well maintained these real world production scars are reflected in the tests.

I hope you are not true at all.

Software like a Database should have an extensive test bench with concurrency tests, all corner cases etc.

I'm not here running the new version on production to tell the maintainer/devs that my 'production unit tests failed'.

What is this even for logic?

I mean there is balance when i write tests for my production software, but my software is used by me. If i would have a library, i would test everything.

And there was some blog post about another database system were they even virtualized the File access to test cases like when the disk controller stops working.

"Everybody has a production system. The lucky ones also have a test system."

Wait - does the AI rewrite the tests too? If so, lol.

The test suite is the result of these years of years of running in production. Every time you fix a bug, you add a non-regression test to ensure you don’t break it again.

In a project like PostgreSQL, those scars are reflected in unit tests demonstrating that they’re fixed. It’d be hard to pass its test suite and not be as robust as the original.

That's not relevant though. All concerns are secondary to security and Rust is the only language with security GUARANTEES. No other language is as secure. Therefore, even the worst Rust rewrite is automatically better than the best work in any other language, because it is the only one with guaranteed security.

If a Rust rewrite of any of your software becomes available and you aren't installing it immediately and without reservation, then you are simply not giving security the priority it both demands and deserves, and that makes you disastrously insecure. This is a serious issue that should be given all priority. There is no room for debate. Your only policies should be security before all else and compliance with those policies must be absolute and without deviation, or all is lost.