Would it be technically possible to transfer tokens to another user directly, without an exchange acting as an intermediate?

My assumption is that this is possible but not supported by default, since there’s no way to guarantee (without using an exchange) that the giver has really deleted the tokens they sent to the receiver, and will not spend them before the receiver can (which would render the receiver’s copy of the tokens unusable).

But assuming someone implements this in software, would it be possible to make direct transfers like this while relying entirely on trust? Or are the tokens somehow tied to the wallet that has first received them?