You skipped the step where gold and silver is used as coinage for everyday transactions. You can in fact own the physical metal and spend it without an IOU.
Although you physically can transact with physical gold. It isn't practical.
Although you physically can transact with physical gold. It isn't practical.