alt Hacker NewsSeems like a non-sequitur? "because math is formalizable", I agree it makes sense to then predict that data-driven AI models will be able to perform well on it. But where is the evidence that software coherence is formalizable? If it were then I would agree, but to me it seems that coherence in a software system relies on a ton of intuitive reasoning and familiarity with company culture, past failures, etc. it's like the least formalizable thing there is.
AI improves over time, sure, but it has differences and advantages and disadvantages over human intelligence. It's just a fundamentally different thing. In the same way calculators are superhuman, and improving, but can't write books. You yourself acknowledge to some degree the uneven nature of that improvement when you referenced formalizability.
In my comment that started the thread, I pointed out that "coherence" in the article is considered valuable because it's a proxy for "ease of modification", but AIs can estimate "ease of modification" directly (by looking at how hard it is for them to make changes to a codebase) and learn from that in a closed loop. No round-trip through humans required, contrary to what the article says.