The point of LVT is right there in the name: it is the land that has value. "Location, location, location" is an old cliche for a reason.

If you let your property go empty and crumble, the Land Value Tax is there to provide the incentive for you to either fix up the property and actually use it, or sell it to someone who will.