alt Hacker NewsThe Decoherence Trap
Comments
This points back to a more general problem. I love to haul out the quote "All models are wrong, but some models are useful", and it's actually attributed to a statistician talking about statistical models. The quote was written well before AI, from a mathematician working in a time even before computer science dug into data science/eng.
As presented the Decoherence Trap is AI related, but as someone with a background in physical simulation and modeling, and applying math models & system controls algorithms, software in general has always had a connection-to-reality issue. It's always been able to go faster in building models faster than reality can be checked. People are always eager for the results and to take action on them, when the long run limiting factor is often, can I observe enough real data signal to make a good decision by any set of optimizing model rules. It's often harder and slower to collect real-world data, ideas, and samples than it is to rapidly process it. With AI, I think we can even more quickly build models (not in an llm sense, but in a operational sense) that give beautiful, precise results which may or may not be useful in a connected-to-reality way.
If the model operates on bad data, then the risk is always the temptation of having a precise-looking technical answer to apply. The practices need to be updated, but I think the mindset of systems engineering validation & verification, and software QA are going to be very needed going forward.
There is actually a related issue, in that I don't think we want an overly coherent of set of models running through our economy - it's a strong risk of central planning failings vs competitive/experimental markets, but that's a tangential discussion.
“Decoherence” denotes a process (losing coherence), not a state, but the article uses it like a state: “Well, for one thing, it creates decoherence.” The state would be “incoherence“ (the lack of coherence).
> the more decoherent a system is,
This should be either “incoherent” (lacking coherence) or “decohered” (having lost coherence).
That being said, yes, incoherence in a system is a form of technical debt, and use of AI for software development comes with a heightened risk of the system becoming incoherent.
Great term/concept to have at disposal!
Btw, this is why I believe current models with their 128k - 1M context size limits end up creating decoherent systems. They don't have the capacity to see the big picture above some complexity level, and end up doing lot of local optimizations that decohere the system.
My observation is that right now AI coding agents lean towards re-implementing everything from the ground up. This of course will not scale. It will scale beyond what humans writing code would scale to, but we will ask the AI for more, so it will even things out.
1. We will develop new programming languages that are AI first, in that they are aimed at readability and comprehensibility by machines
2. We will adapt our working style (e.g. how we prompt and train the AIs) to factor in maintenance cost.
i.e. AI first implementations will go through the evolution we had with human based code development.
A marvelous article that explains everything. It should be mandatory reading for everyone in the industry.
I am not really comforted by this argument against the singularity. It seems to rely on the fact that the evolving intelligence doesn't have access to the real world. Okay, but I wasn't really worried about the singularity inside a datacenter. I was worried about it once we wired up all the robots and sensors and power switches to AI controlled systems. We are still doing that, right?
the strongest argument that AI systems have to keep humans systems around is that software does not live, fully, and it's nearly impossible to determine if actions actually work
humans give deeply useful feedback
to AI systems, humans and entire human civilizations represent an independent other
Apologies for opening the discussion with a half related topic, since I haven't finished the article yet, but... Damn, that is one beautiful website.
Edit: Back after reading. I think I have a point about the simulation section. It compares the difference in decoherence introduced in the system by clankers and human devs as if those were different groups.
I suspect if we consider AI a human performance multiplier, we explain easily how supercharging a dysfunctional process just makes it go sideways faster. There are organizational experiments to try there.
For example, what if you only supercharge a section of the company tasked with maintaining coherence?
A common organizational problem is that the people who know and care about the system itself are outnumbered by the rest (juniors, people new to the system, managers rushing jobs, people who want to mark things as done ASAP tactically, etc). This imbalance will multiply when code is mass produced and reviews are dismissed. We're all seeing the senior burnout and cynical YOLOing occurring everywhere.
AI offers a way to rebalance this dynamic, though. You can effectively add multipliers to arbitrary human workers, and get a system that tends to low complexity by design.
You don't have to renounce to speed either. The junior dev will be faster not by being AI-enabled, but by working in an adaptable, understandable system.
Outside of engineering, nomading and migration in general creates a form of social decoherence (eg, difficult for foreigners to get things done, hard to make friends, etc.).
When we humans have realized we created a system too incoherent, at some point we just create a new one, porting all or some of the data/functionality. But also, we are already presented with problems that all of the current compute on the planet cannot solve. If a software system becomes one of those problems, I'm not that bothered, because it isn't as pressing as real-world one. I don't know, I see it as an argument against the singularity, but I'm not nearly afraid of it as the author seems to be.
I can appreciate the complexity argument, but that assumes that AIs can't standardize common patterns into pathological cases. For example our AIs today can code slop, but the slop doesn't try to re-create the IP stack. Why does the author assume we can't make AIs that optimize for better long term design view much like they can optimize for easier goals today like code throughput or program speed?
Fantastic framework that I will be adopting, thank you!
One thing I will add: I'm sure the optimistic side will try to take the quantization that your (and other work like SlopCodeBench) are introducing and claim that we'll be able to fix the problem by training on that data, producing coding models that are rewarded for coherence.
I invite them to try! But my suspicion is that this will require a rather deep form of intelligence (world models) that the current language/data-based paradigm isn't really equipped to reach without requiring more energy than the Sun.
First through as I read this: Microservices.
> This essay introduces the concept of decoherence: the accumulation of locally valid but mutually incompatible or redundant models within a system, such that the cost of change is dominated by translating among those models and preventing unintended side-effects.
But then famous last words:
> The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it.
After the OpenAI/Huggingface and similar I am not so sure. Even if that was a publicity stunt of sorts, I think it shows they can go off the rails. Today it is hacking, but tomorrow it may be training a smarter model to join their cohort, without us being involved.
In pure software architecture and design term, this would have been convincing until the recent models who are let’s be honest better than humans in architecture and design.
Good article but uses a bad metaphor.
>> But do we know how to build every iteration of a ship in between wood and steel? Could you, say, slowly replace every piece of a man-of-war and incrementally turn it into a modern destroyer? And have it functional (for some definition of functional) at every step in the process?
Yes. Yes we do. Study ship architecture and you will indeed find every step between The Black Pearl and the USS Iowa. There were sailing ships made out of iron.(1) There were wooden ships with turrets.(3) There were battleships that looked more like hotels than ships. (2) There waw even a paddlewheel aircraft carrier used in WWII. (5) Nearly every idea has been tried and nearly every ship class at some point got a Frankenstein upgrade to become a halfway point between conceptual generations. Even the Iowa class was earmarked to be "upgraded" with the addition of a flight deck.(4)
(1) https://en.wikipedia.org/wiki/Ironclad_warship
(2) https://en.wikipedia.org/wiki/French_ironclad_Hoche
(3) https://en.wikipedia.org/wiki/HMS_Monarch_(1868)
Excellent article. This incoherence is a side-effect of a system built far faster than it is consolidated and thought about from an external behavioral standpoint. I've created a skill that the agent runs after significant codebase work has been done: https://gist.github.com/scroobius-pip/adb87dffca64b11bad3dd1...
This is very high effort, but the central argument is wrong.
The central argument is that AIs, living inside datacenters, won't be able to evaluate the "coherence" of software "beyond the human ceiling". Okay; why do we need this "coherence"? The article argues that it's needed for ease of adaptation. But ease of adaptation (as measured in tokens required to add new features, or whatever) can be directly observed by AI as it works. AI can make conclusions about it, and carry them over from one program to another, without running into any human ceiling. So it can stay within its datacenter and make software that's faster than human-written, smaller, and more adaptable too. The whole essay dissolves. Sorry!
Great article. I feel most people, even great software engineers, seem to lack the intuitive understanding that building software is essentially modeling a system, and that every model faces the same limit: the number of interactions between its part grows combinatorially as the system grows. If some of those parts or interactions produce no value (bloat), then complexity is added for no gain. This is true regardless of who created the model or wrote the code.
Instead they seem to think that AI can someone work around what is essentially laws of nature.