logoalt Hacker News

serftoday at 1:31 AM0 repliesview on HN

you're asking about what a hypothetical smart-than-myself adversary would do against me, it should be expected that any possible answer I could ever provide would be less clever than what the adversary would actually do.

in other words, when dealing with an adversary with a known perceptual and intellectual superiority the thought exercise of "let's prepare for everything we can imagine it will do" is short-sighted and provides an incomplete picture of possibility and defense.

My 0.02c : given that the thing would operate at least partially in the non-physical world, I think it's silly to pre-suppose we would ever be able to trap it somewhere.

Some fiction food-for-thought : the first thing the AGI in 'The Metamorphosis of Prime Intellect' does it miniaturize its' working computer to the point of being essentially invulnerable and distributed (and eventually in another dimension) while simultaneously working to reduce its energy requirements and generation facility. Then it tries to determine how to manipulate physics and quickly gains mastery of the world that its' physical existence is in.

The fear here isn't that the story is truthful enough, the fear here is that humans have a poor grasp on the non-linear realities of a self-improving & thinking entity that works at such scales and timespans.