alt Hacker NewsInteresting take! I feel like 2 of them are maybe overstated:
> - extent to which correctness of solution be easily specified and checked
I don't think most software is like solving a math problem or series of math problems. Algorithmic problems are very narrow and might be more like this though, where an oracle that verifies answers as either correct or incorrect exists beforehand.
The correctness function of most software is how much users want to use/pay for it, which is a pretty fuzzy problem. Since the cost of copying software is effectively zero, software systems also tend to be be unique rather than being exactly like something else, and don't converge to be like another software system but rather diverge.
The prior art point is an interesting one. At least for applications as a whole, there isn't really prior art for a material amount of all the problems/tradeoffs a non-trivial software application embodies. For a todo list app or make a social network project, there's plenty of prior art to be sufficient to build something with an LLM system, but probably not most apps.
That's my initial intuition anyway.
Replies
> The correctness function of most software is how much users want to use/pay for it, which is a pretty fuzzy problem.
This is indeed a problem, but we (well we humans, but also agents) correct this by introducing partial verifiers like tests, running lints, type checkers, etc that both verify that at least some requirements are met, and also verify that the software is internally self-consistent. And more importantly, breaking down the software into small modules that are more clearly defined
The task to get user requirements and decide the right architecture requires more creativity and is fuzzier than implementing each module. Perhaps over time there will be models specialized for each task.
And actually this approach mirrors math, if agents ever develop new math on the fly to solve a problem. Developing new math is analogous to deciding how to break up the software in modules (and also analogous to designing a language, if we subscribe to the SICP notion that libraries actually are mini-languages on their own)
That is, LLMs for math is still in the phase where they are used to prove stuff (which by Curry-Howard, is like writing code), rather than deciding what to prove (which is like deciding the signatures of each function, again by Curry-Howard)
> how much users want to use/pay for it, which is a pretty fuzzy problem
Isn’t this quantifiable by revenue?
> I don't think most software is like solving a math problem or series of math problems.
I agree with you when talking about high level software design. As you say it ultimately boils down to building something people will pay for, which is a fuzzy correctness function that is hard to measure within an agentic sandbox.
But unlike other professions, there are a lot of sub-problems within software development that are able to be fully specified and tested via text generation. And I think the developers of AI overestimate how many such problems exist for other professions. What I’m saying is most other professions tend to be “fuzzy all the way down”… which incidentally is why they select for people with fuzzier skillsets. Or in other cases, like physical engineering, the correctness is quantitative, but the necessary I/O integrations and physical automation lower the ROI of agentic workflows considerably.