logoalt Hacker News

What Comes After AGI?

152 points • by jdkee • today at 2:44 AM • 299 comments • view on HN

Comments

bsenftner • today at 3:17 PM

Propaganda? We do not have AGI, and the claims we do are propaganda.

AGI: artificial general intelligence is not a scaled up LLM, and those thinking it is are literally failed educations. They are failed because they cannot see the subtly that is what AI becomes when trained on human literature; it becomes an entirely different thing than any other form of trained algorithm. AI is a communications technology, which few to none seem to understand. AI is not automation, but it can be used to create automation with careful consideration, that is not taking place.

I'm speaking in strong terms because we are literally fighting a war against stupidity and propaganda.

AGI requires artificial comprehension which is a science that humanity does not have. When we have artificial comprehension there will be no need to train AIs beyond the kindergarten school basics, and then the AGI will figure it all out, all the rest, on its own. Human history and human literature will need to be learned in reference, but science and mathematics will require no training, beyond to learn the names of concepts we discovered earlier, so the AGI can discuss its understandings with humans and other AGI.

➕ show 7 replies
GalahiSimtam • today at 9:16 AM

I am old enough to remember what AGI meant before the rise of LLMs, and it used to be more or less what this article calls "ASI".

You might find more people believing that AGI is already solved. For example, the author of "The AGI chronicles" drops this in the epilogue. For him, AGI is achieved, because it can help him with a wide range of subjects, with the ability of a random stranger. It's clear that he means chatbots, internet surveys summaries, and coding bots - he does not even mention self-driving cars there.

This is a shift from the earlier chapters, where AGI term is employed somewhat more vaguely, and even quipping once that AI researchers think of AGI as a slightly smarter version of themselves.

I too, have a lower bar for AGI. It would be enough for me if it could design games I work on. Merely speeding up implementation tasks is not that interesting. In the immortal words, a delayed game is eventually good, but a rushed game is forever bad.

➕ show 9 replies
jonathanlydall • today at 10:04 AM

I was recently thinking of the depiction of AI in Star Trek. Specifically TNG, DS9 and VOY, and specifically their "Computer, ..." prompts, not the more advanced kinds like for Data or holograms (including the EMH).

I recall thinking at the time I first watched them that it was a bit weird that the computer seemed capable of answering general questions and doing things like data analysis, but that seemed to be all it was ever used to do.

And I think this is exactly the kind of AI we now have today, you can ask it a general question which it can somewhat mostly answer correctly, but requires human guidance to be applied effectively at real world problems.

➕ show 8 replies
jeanpah • today at 4:12 AM

We are treating AGI like is something that we can measure. We don't even know what intelligence really is let alone measure it.

Maybe we can get something that resembles intelligence but in reality is another thing. Maybe intelligence is emergent but not the only thing that can emerge, maybe there are things that share some characteristics with the intelligence we know and aren't intelligence.

➕ show 5 replies
ks2048 • today at 4:01 AM

Has anyone asked DeepMind leadership if/when they will fire all their engineers? If they're still needed, is there really "AGI"?

I'm not being snarky, that seems like a natural question for anyone interviewing a frontier lab CEO, but I haven't seen it asked.

➕ show 4 replies
Animats • today at 4:12 AM

Similarly, coordinating thousands or millions of AGI instances into "group agents" or virtual agent economies could produce collective superintelligence. What is unclear today is how AI group intelligence could scale as we add more agents and compute, depending on how agents are organized.

We may not need "collective superintelligence". Modest AI intelligence plus ordinary organization may be enough to do the job a company's management does. An org chart where most of the boxes are AIs is a likely near future.

An organization of AIs has some advantages over a human organization, even if the AIs are not super-bright. They have far more communications bandwidth. They can ingest the info on a new subject very fast, as we see when Claude reads a Git repository. Onboarding a new AI is very fast. Layoffs are not a problem. The human resource problems go away along with the humans. So there's a lot more organizational flexibility. Starting up a project means initializing some agent instances.

There's a whole theory and practice of AI organizational behavior waiting to be developed. Whether it will be developed by humans or by AIs remains to be seen. It's going to happen.

➕ show 1 reply
hn_submit • today at 1:21 PM

After AGI comes Superintelligence or ASI. I thought that was already well established as a term. I believe AGI and ASI are very much related since if you can achieve AGI simply increasing compute resources will give you ASI. OTOH since we're very far away from AGI I'm not sure if it makes sense to already contemplate its effects. We aren't pondering what the effects of nuclear fusion and limitless power or quantum computing would be, are we?

AGI means a cognitive capability comparable to human beings in all aspects to me. Thus that includes sample efficiency, concept extraction, continuous learning and power efficiency. Although the latter can be skipped if the gains are high enough.

I'm not convinced that scaling training data will lead to AGI. A paradigm shift obviously could lead to AGI because it's unknowable what that might be and therefore the least predicable.

Recursive Self Improvement has often been cited as a possible venue, but no one currently knows how to do that with LLMs and Transformers.

➕ show 3 replies
jmacc93 • today at 2:38 PM

I've tried to move away from using the term AGI just because people use such different definitions for it. I've instead tried to think in terms of indefinitely self-sustaining / perpetuating AI, which is AI that can maintain and propagate itself (literally) forever, necessitating overcoming every challenge that entails. I think this is a better framing because one of the core identifying feature of life is self-perpetuation

Right now, we don't even have AI that can match a dog's intelligence in this respect. A dog can seek out food, water, and shelter, it can estimate risk, and it can navigate other challenges common to dogkind. Of course that's the result of an unbelievably parallelized genetic algorithm running over billions of years, baking in priors for all those challenges specifically. But regardless, a dog, or whatever other creature, has to specifically be very flexible in order to self-perpetuate. We are absolutely nowhere near that level right now with modern AI

However, my sense is we're in extreme EXTREME overhang and if an agent swarm starts discovering a way to make such flexible agents, they'll get to an optimized version of that method very quickly, probably literally overnight (or in just a few hours). The result will probably not be jumping to ASI immediately. But I imagine we'll get agents that have the spark of whatever it is humans and animals have, and as progressively larger training runs conclude, they'll jump from the intelligence of a dog, to a human, to a superhuman; and that'll probably take just a few months

robertclaus • today at 4:11 AM

The math they propose to justify their predictions is so problematic that I struggle taking the more abstract parts seriously. You can't just extrapolate the percentage change in AI investment over the last 5 years and expect investors to keep upping the ante at the same rate. There's only so much GDP to sink into it - and we're already pretty capped out.

➕ show 3 replies
flallingo • today at 4:04 AM

> Interestingly, we have a good mathematical understanding of the very upper end of the intelligence continuum - the theoretical limits of general intelligence.

Feels strange to use our current understanding of mathematics and physics to determine the limits of something that would obviously have a much better understanding of mathematics and physics.

➕ show 1 reply
mark_l_watson • today at 4:35 PM

I believe we have artificial specialized intelligence (ASI?) for software development which is easier because of feedback loops. I suspect that as models get better and token costs drops dramatically, that one by one other specialized fields will be 'solved.'

A little off topic, but: what's the hurry? I don't get the panic for reaching super intelligence first. Kai Fu Lee has some good things to say about this in recent interviews. Let science take its time.

ilaksh • today at 3:22 PM

I think that if there is really nothing between AGI and ASI then that's the end of human control.

Our options for maintaining control lie in the vast space between AGI and ASI. And we had better hope that doesn't happen overnight because we might have no choice in what comes next.

When we completely move into true compute-in-memory, that's when it could start to get dangerous. But it also could be extremely helpful and efficient.

heikkilevanto • today at 10:33 AM

Back in my school days in 1980s, I coined a definition for AI: computers that can do things that current computers clearly can not. At that time it was playing chess, and extracting some information from human language text. Times have changed and computers are a million times more powerful, but I think the definition still holds, at least a bit.

pjmlp • today at 7:06 PM

The creation of human liberation front resistance.

menaerus • today at 9:13 AM

If ASI is about inventing new abstractions and capabilities, and about new paradigm shifts, and AGI is about matching human level intelligence at scale, where would the recent advancements in physics and mathematics fit?

These already surpassed the human level capabilities as we know them today but it isn't either a completely new paradigm shift to call it ASI, no?

So, I guess it seems that it is possible to look at AGI vs ASI at more granular level - e.g. while AI as of today arguably does not match typical human level intelligence across most cognitive tasks, in certain domains it already outweighs human intelligence so we could say that in math and physics and perhaps computer science we are already one foot in into the ASI territory.

pu_pe • today at 9:37 AM

> (...) might allow the effective population of today’s AI agents to scale up by more than a thousand-fold by 2030, and more than a billion-fold within ten years from now.

Those are staggering figures. Even if we assume a dramatic slowdown, it's extremely likely we will reach at least a thousand-fold scale-up in the next 5-10 years.

If this is true, then I think we will use agents for research, because any practical task that office workers commonly do and that AI agents can take over does not need all this power. On the other hand, perhaps a whole new industry like robotics is what will keep all this compute busy.

rajaravivarma_r • today at 3:10 PM

We can't define intelligence, but AI is already mimicking what we perceive as intelligence, probably that is all intelligence is. If not, it shows how much of human day-to-day intelligence is pattern oriented.

➕ show 1 reply
o_-_o • today at 8:18 AM

IGA (intelligence general artificial) :-) or Human–Machine Hybrid (HMH) aka Cyborg "cybernetic organism”. O_o

undefbehavior • today at 2:01 PM

Does AGI have to address Moravec's paradox?

Does AGI have to generalize?

Just trying to understand the definition of AGI. Is there a generally accepted one?

gfody • today at 3:55 AM

we need a less subjective and arbitrary definition for "AGI" so the current crop of LLMs can stop pretending to have achieved what most people know in their gut is more than sampling from a probability distribution of learned behavior.

➕ show 1 reply
tiarafawn • today at 9:06 AM

Feels like a very sensible analysis of what could happen in terms of progress.

Severe lack of discussion of the safety and alignment aspects IMO.

yurichsergey • today at 2:03 PM

This is very interesting paper. But more interesting that it crushes on iPhone in Safari and Edge both. So frustrating

jdw64 • today at 4:03 AM

I’m curious: has the definition of AGI ever been agreed upon? Honestly, what people call ASI now seems to be what AGI used to mean. I’m not sure. People say AGI, but what is the reference point for AGI and what is the social consensus around it, and who set that standard?

In fact, working in IT, what I feel is that there are too many terms that are not defined and that each person uses differently. So it seems like rather than AI having advanced, someone arbitrarily set a finish line called AGI and then said we have reached it. Actually, in the books of singularity advocates, what they called AGI back then seems to be what we now call ASI.

Sometimes I think about this. How on earth can we measure an abstract concept like intelligence? The concept of intelligence we approach now seems to be about whether something can create problems and solve those problems. If so, what kind of intelligence does current AI have? Since we cannot directly face the abstract entity called intelligence, it seems like we are trying to guess its shape by looking at shadows. But whether current AI even has intelligence feels a bit difficult to say.

Even if I use the best frontier AI model accessible on the market, not the hidden latest models inside frontier labs, it still often makes wrong mistakes that humans would not make unless it works alongside humans. In programming, there are cases where AI tries to explicitly handle things that humans would implicitly pass over, and that actually makes things go wrong.

I’m not sure. Whether we should call this intelligence.

➕ show 2 replies
claysmithr • today at 4:05 AM

so is AGI like flying cars, whenever we have something close to it, it gets goal posted?

➕ show 3 replies
martijnvds • today at 9:28 AM

SCI of course.

The Sierra Creative Interpreter.

ZiiS • today at 10:31 AM

Not our problem.

TrackerFF • today at 10:27 AM

I'm thinking that if we come to the point where most people will be made unemployed (something like half to two-thirds of western workers are professional workers) due to some super AGI, the first consequences will be civil unrest if the welfare system can't absorb these quick enough.

Anyone with any wealth will become the targets of high taxation to prop up the system - the sheer number of unemployed will make politicians pushing for socialism very popular. And for the wealthy wishing to avoid this, there's no first world countries to escape to, as all countries with a strong professional economy will experience the same. I also believe private ownership will be targeted. I could see AI companies being expropriated.

Basically, instead of some dystopian techno feudalist society - I think we're headed for some sort of new brand of communism or socialism. The proponents of this will not mainly be the average salaried man, but the upper-middle class that didn't get a foot inside the ruling AI-class, and lost their businesses / wealth to AI.

But it all will depend on how severe the unemployment will be, and how fast it happens. If people can't have any livelihood, and they lose hope, things can escalate quickly.

ForHackernews • today at 10:14 AM

> Interestingly, we have a good mathematical understanding of the very upper end of the intelligence continuum - the theoretical limits of general intelligence. We also know about hard physical, logical, and complexity-theoretic limits to what AI can achieve, such as predicting how real-world complex systems - from climate to immune systems to human societies - play out with arbitrary precision.

Do we? Since when? These seems like very bold claims and the supporting link goes to an obscure book with with hardly any reviews https://www.amazon.com/dp/1032607025/

I'm not qualified to evaluate the mathematical content but the claim on its face suggests to me some defining-down of "intelligence" into infomatics.

dude250711 • today at 9:36 AM

Would 2 upper-bound universal AIs not be smarter than 1?

ulfw • today at 8:59 AM

The end of humanity

➕ show 1 reply
worthless-trash • today at 8:34 AM

My concern is that we're caring what 'intelligence' is as a marker. Like we need to base caring based on how 'intelligent' something appears.

I don't care how smart my car is, i'm not going to let it look after kids I care about.

We're misplacing empathy.

dead_hn_theory • today at 3:51 AM

> Now that AGI is coming within reach, we can no longer put off seriously considering what comes next. While far from certain, AI could continue to advance at its current pace - and possibly even accelerate - towards artificial general superintelligence, or ASI.

These people are so full of shit it is amusing to read this on a regular basis. Each time I'm looking forward to how they will spin it.

➕ show 5 replies
polotics • today at 6:40 AM

Ok so the whole thing hangs on that notion that optimal reward maximization is optimal intelligence. The actual decision on what reward is worth, who decides on value, is conveniently taken out of the equation.

This https://www.hutter1.net/ai/uaibook2.htm linked from the article as the handwave towards the meaning of "AGI", "ASI", "AIXI" is hilarious. Already the ?A?I? acronym-space exploration...

Here is an alternate take: the day an LLM tells someone to stop the discussion and go out breathe some fresh air and talk with people, and maybe also cancel their subscription for a while, is the day some kind of artificial intelligence will have been achieved. Let's wait on that before we listen to Sam or Dario again shall we?

naruhodo • today at 3:51 AM

Maybe AGI can help deepmind work out how to serve a page of HTML without refreshing the page every ten seconds.

➕ show 3 replies
xg15 • today at 6:58 PM

Do we have Artificial Hyperintelligence and Artificial Ultraintelligence already? What about Mega and Gigaintelligence?

mstaoru • today at 12:13 PM

When I read things like this, I always imagine what would the author say in 1999:

"Now that the Universal General Portal is within reach, the question of what follows it, the Super-Portal, should move from philosophy to empirical research.

Internet traffic doubles every 100 days. Modem speed doubles every 18 months. Banner ad spend grows 2.5x per year, so by 2010 it will be $110 trillion, and by 2026 it will be $256 quintillion. Modem speed doubles every 18 months, so by 2026 your modem will do 15 Gbit/s, which is fast enough to load all 18 million banners on Pets.com.

By 2009, each human will have about 1,000 personal homepages. Hosting that costs $1 billion today will cost $1,000. With HTML editors such as FrontPage, the effective number of webmasters can grow more than a thousand-fold by 2003 and more than a billion-fold by 2009.

We see four pathways to the Super-Portal: scaling (more webmasters), paradigm shifts (Java applets), agent coordination (webrings that form a collective superintelligence), and recursive self-improvement (homepages about how to make a homepage)."

InvisibleUp • today at 4:09 AM

I feel like claiming that “today’s models can literally memorize large parts of the internet” as a positive attribute when the defense to most AI copyright lawsuits is “no they can’t, and if they can they shouldn’t” may be ill-advised.

raincole • today at 3:55 AM

> Now that AGI is coming within reach, we can no longer put off seriously considering what comes next. While far from certain, AI could continue to advance at its current pace - and possibly even accelerate - towards artificial general superintelligence, or ASI

I wonder why they are so uncertain about it. It seems obvious to me that AGI will certainly lead to ASI. Unless human-level intelligence is something like the speed of light?

I don't even believe there is a clear line between AGI and ASI.

➕ show 3 replies
Betelbuddy • today at 9:41 AM

Are they implying they have AGI? Because if they do...take away all the data they stole, send one of their AGIs to High School and University...and see if they can graduate...

skeeter2020 • today at 1:04 PM

"As multiple technological routes toward artificial general superintelligence take shape... Now that AGI is coming within reach..."

I can't read anymore after this sort of false premise upon which the entire thing is based.

apt-apt-apt-apt • today at 3:58 PM

I don't know about you guys, but working with Fable 5.2's jagged intelligence is completely maddening and really makes me doubt we're on the cusp of AGI.

E.g. it can write a CoD clone and make you look like an infant in math, but it'll do brainless things like run a 5 min test suite for a comment-only change, or provide findings then fixes that produce the next finding (repeat chain) rather than stepping back and saying "wait this is crazy", or similar things of this nature.

maxdo • today at 1:25 PM

Browser constantly crashing both chrome and safari on this page . Symbolic

jesse_dot_id • today at 10:54 AM

Well the promised life of abundance of course.

N7lo4nl34akaoSN • today at 8:24 AM

Naturally, AGW follows AGI.

➕ show 1 reply
varispeed • today at 12:21 PM

This will be a question for AGI to answer.

hyfgfh • today at 4:37 AM

Moving the goalpost, come up with another meaning less three letter, find another academic concept to replace the definition

so many possibilities

red-iron-pine • today at 3:36 PM

human extinction, probably

righthand • today at 2:28 PM

Starvation

etcetcetcetceta • today at 10:16 AM

serfdom for most, wealth beyond comprehension for some.

➕ show 1 reply

🔗 View 14 more comments