alt Hacker NewsIt's not me who's "redefining knowledge to my convenience". You weirdly succumb to "them" doing that and come to wrong conclusions.
Merely claimed "descriptions of reality" lack the crucial property of truth. The description has to actually fit reality, or it's mere fantasy. It has to be exhaustive, or it remains a facade, leaving you blind to what lies beneath.
When you don't actually know whether it does any of that, you're merely a believer, entirely dependent on your "oracle" and its slave in effect.
Not caring about nuance leaves you open to exploitation, among other things.
The oracle is infalliable in the toy scenario. That was asserted as an axiom. To us who are god's eye observers of the toy universe, we already know that whatever the oracle says necessarily fits reality in that universe.
But I understand what you mean. Suppose I am Archimedes in there. I don't have access to the axiom, so I don't know if what the oracle told me is true until I can verify it myself. So verify I will. This is what is being done with AI-written proofs IRL. Humans may not be able to internalize the proofs, but we can internalize the programs which verify the proofs. So we do know the proofs are correct, or rather, as correct as the verifier programs.