Some of this post reminds me of a story I heard long ago from someone who had worked at a HW/SW company. They’d transferred an engineer from the ASIC design team to the OS kernel team, though he’d never been on a software team before. After a while the manager called him in for the following conversation:

Manager: You’re doing amazing work — zero bugs in production! I’d like you to mentor the other SWEs on how to get their bug count down too.

Engineer: We’re allowed to have bugs?

For large projects like this I think a hierarchical division of labor also helps.

If you first carefully define the overall architecture and thus individual high level components of the system, then you know which of those components are mission critical and which are commodity. Mission critical would be anything ensuring ACID, etc. That way, no matter what you farm out to LLMs, you can keep the majority of limited human focus on the far fewer mission critical components. If tests end up not being robust enough to catch all issues, at least they'll be isolated to commodity code where damage is limited to things like DoS, etc, and not code that could cause data loss.

I also think it's important to first define the _contracts_ on and between each of these components, and derive tests from those contracts. Partly because contracts more succinct and easier to reason about. And partly because Rust provides many tools to enforce contracts at compile time, reducing the need for tests (which themselves could end up subtly flawed). Contracts can be enforced through typing, private vs public APIs, etc. Newtypes are _incredibly_ powerful for both enforcing contracts and making footguns much less likely.

Oh, I just posted a similar comment elsewhere in the thread.

https://news.ycombinator.com/item?id=48856535

Though beyond testing, I think there will be increasing focus on proofs of correctness. (Testing can only show the presence of bugs, not the absence. —Dijkstra)

At any rate, it's never been this cheap to produce the proof of correctness of a program, or on the other hand, to produce an exploit for an incorrect program.

And what happens when your "tests" are also vibecoded. Right now all of these houses of cards reset on human-written tests. What happens without them?

> reviewing the testing/fuzzing process

I've got insanely good at designing testing oracles over the last year for exactly this reason.

I've ported some extremely finicky software between languages that it would have been borderline abusive to have a human do.

Codex 5.3 and later for those interested.