As the saying goes: “the dose makes the poison”. Many (perhaps most) drugs cause harm at the wrong dosage.
Depending on your definition of poison of course, but at least for LSD and magic mushrooms, all that's known is a lower bound for their biological toxicity. For LSD IIRC it's somewhere in the 30mg (!) range, for mushrooms it's more than can comfortably fit a stomach. I'm afraid I don't have sources handy though.
That said, the state of mind must be so wild that "is it poisonous in this specific/usual sense" is probably not the right thing to discuss.
So for mold, low enough doses aren't poison? Can't you just say this about anything. If you have poison in a low enough dose it's not poison. So it means jsut about nothing to say alcohol isn't poisonous because you can have a small dose of it.
> Many (perhaps most) drugs cause harm at the wrong dosage.
Not only drugs, almost everything is bad at the wrong dosage. Case in point: water.