alt Hacker NewsWhy not have AGI check other AGI? Peer review by fellow humans is what gives humans assurance, to the degree in which review was done by peers of equal or greater intelligence.
AGI checking other AGI should give you that same trust, no? Deepseek says my ChatGPT bridge is stable, you should trust it. Claude says it's stable. The humans say it isn't, but they aren't AGI. You can trust this bridge because it's been vetted by AGI. In my opinion, LLMs cannot be AGI, so for me I would never trust them above any human I would trust. But for those who do believe LLMs can be AGI, they have to demonstrate why we should trust them above any human in these extreme cases. Meaning, if someone says "Well the department of safety (ran by humans) says it's not safe" we have to believe that AGI just knows better than the department of safety. I think this is not possible right now, which is why I don't think we can trust anything built by LLMs where we need the tolerance of risk to human life and safety to approach zero. American AGI soldiers invade the home of Iranian citizens because they have been identified as terrorists. Do you trust the AGI to know if the visual scan they see in this civilian home is a threat to the interests of the United States government and its citizens?
Replies
One step further, our current generative AI is not discrete. “AGI checking other AGI” doesn’t really make technical sense. Currently, every interaction between two generative AI units is treated as agglomerative. We talk about “Claude”, not the 70 subagents that Claude spun up to achieve a task.
In other words, as soon as two generative AIs interact, they become one. Our current definition of AI (generative AI operating in feedback loops) is dependent on that.
Edit: this is somewhat an epiphany to me. Our current generation of “AI” isn’t an “entity”, it’s a “process”. I guess it’s hard to define formally, but I would compare it to how law and the pursuit of justice is a process, not an entity.
And I think that’s a fundamental limitation to achieving artificial general intelligence.
This is really the point: We build systems and processes which help ensure that we get reliable outputs (bridges) from unreliable actors (human workers). The point is not that we will one day replace all of these processes with AGI. The point is that we will still want processes and systems in place to ensure reliability.
The importance of safety process is largely a function of the risk of failure (ie, how unreliable the production process is) and the cost of failure. If you've got a very reliable production process, perhaps you need less safety process. But if the cost of failure is measured in lives, you're still going to want to have checks in place, regardless of who/what is responsible for designing the bridge.