It is not "THE" issue, it's barely even "an issue". The amount of radioactive material produced by a fission plant, and the form in which it comes, makes it trivial to store relatively safely - certainly much, much easier than the CO2 waste that most of our other energy generation solutions emit.
Also, the biggest issues with nuclear power are (1) the risk of catastrophic meltdowns, (2) the risk of using it as cover for nuclear armament, (3) the massive capital expenditure to create a plant, and (4) the amount of water needed for cooling and running the plant. All of these make the problem of taking some radioactive rocks and burying them trivial in comparison.
Do I remember correctly that modern thorium-based reactor designs mitigate at least #1 and #2?