After AGI comes Superintelligence or ASI. I thought that was already well established as a term. I believe AGI and ASI are very much related since if you can achieve AGI simply increasing compute resources will give you ASI. OTOH since we're very far away from AGI I'm not sure if it makes sense to already contemplate its effects. We aren't pondering what the effects of nuclear fusion and limitless power or quantum computing would be, are we?
AGI means a cognitive capability comparable to human beings in all aspects to me. Thus that includes sample efficiency, concept extraction, continuous learning and power efficiency. Although the latter can be skipped if the gains are high enough.
I'm not convinced that scaling training data will lead to AGI. A paradigm shift obviously could lead to AGI because it's unknowable what that might be and therefore the least predicable.
Recursive Self Improvement has often been cited as a possible venue, but no one currently knows how to do that with LLMs and Transformers.
> We aren't pondering what the effects of nuclear fusion and limitless power or quantum computing would be, are we?
What? All the time!
We've been seeing discussions on post-quantum encryption for a decade or two. The effects of nuclear fusion have been in the background of countless scifi stories, the idea of an extreme amount of abundant energy unlocks 100s of "that can't happen due to energy density" dead-end what-ifs.
Additionally, we're pondering the effects of AGI/SGI because a handful of extremely rich people have been telling us we'll all be out of all our jobs in x years, for almost 6 years now.
They've literally been sitting in press releases telling us "Enjoy that comfort because you won't be able to feed your family when you're outsourced. Your work and career will be worthless, and we're leading the way towards returning you to peasant life!"
Regardless of how out-of-reach AGI is right now, people will continue to work on it unless we end up in some Butlerian jihad situation. Just like how fusion is still out of reach, but far closer. They've got fusion reactors running for over a minute now! 10 years ago they could barely get the metaphorical engine to turn over.
Why wouldn't we ponder the effects? Do you walk through doors without consideration of what's on the other side?
> AGI means a cognitive capability comparable to human beings in all aspects to me
So a medical doctor and a structural engineer can never be considered general intelligence separately?
What we have is spiky ASI. It's really good at a huge number of things that nobody can be good at simultaneously. It also sucks at some stuff that's really obvious and easy for us.
Intelligence is grown and it will continue to grow. Thinking intelligence can be designed is walking into the same dead end many people before you already have.
Edit: They have edited their comment beyond what I responded to now.