logoalt Hacker News

alecst • today at 8:14 PM • 0 replies • view on HN

     In the 1960s there was recognition of some of the implications of
     human intelligence. I. J. Good wrote [10]:
     
     Let an ultraintelligent machine be defined as a machine
     that can far surpass all the intellectual activities of any
     any man however clever.  Since the design of machines is one of
     these intellectual activities, an ultraintelligent machine could
     design even better machines; there would then unquestionably
     be an "intelligence explosion," and the intelligence of man
     would be left far behind.  Thus the first ultraintelligent
     machine is the _last_ invention that man need ever make, 
     provided that the machine is docile enough to tell us how to
     keep it under control.
     ...
     It is more probable than not that, within the twentieth century,
     an ultraintelligent machine will be built and that it will be
     the last invention that man need make.
     
     Good has captured the essence of the runaway, but does not pursue
     its most disturbing consequences. Any intelligent machine of the sort
     he describes would not be humankind's "tool" -- any more than humans
     are the tools of rabbits or robins or chimpanzees. 
-Vernor Vigne

https://edoras.sdsu.edu/~vinge/misc/singularity.html