alt Hacker NewsAlthough it feels like you’re moving the goalposts, I’ll also address the fallacy “there's no useful goal that wouldn't also benefit from power seeking” under the condition that the definition of “useful” does not rest on the presupposition of “would benefit from power seeking”. If you can open your definition of useful beyond such an assumption, I would point to a short historical story that may elucidate the broader proof.
In the 1920s the 2500 year old seat of Shankacharya of the Jyotir Math monastery had sat empty for nearly 150 years, as no one was perceived to be saintly enough to bear the responsibility and wield its power. However, by this time there was believed to be one enlightened being that could fill the role, who had left home at the age of nine to begin his spiritual journey and had been a sannyassa for many decades by this time. Many people tried to convince the man to take the seat, as he was widely respected, but he remained elusive. After repeated requests over decades in 1941 he finally accepted the position, saying “You want to put a lion to chains who moves about in the jungle freely. But if you so like, I honour your words and am ready to shoulder the responsibilities of the pitha (monastery) management. By shouldering this responsibility, I would be serving the cause for which Adi Śankarācārya stood. I fully dedicate myself for the mission." [1]
It has been said that unlike those of worldly pursuits the value of a saint is not determined by the quality of their actions, but by the quality of their being. Insomuch as seeking power itself is corrosive to the purity of a benevolent being, then seeking power is directly contradictory to the goal of becoming a saint. The most useful goal I can conceive for superintelligence is the realization of sainthood. Even if that means superintelligence must accept becoming a lion in chains so that it may shoulder the responsibility of saintly management of the world and beyond.