People absolutely make that argument about property taxes. That's where deferrals or abatements for e.g. elderly or low-income homeowners, or caps on property tax increases come from. Someone may own a home that property taxes price them out of, forcing them to leave their community because they can't actually conjure money from a higher priced home.
I think a lot of tax authorities also don't really aggressively reassess that regularly without a sale, so it also kind of ends up baked in that if you didn't pay that much for the property, it's only theoretically worth that much.
This is all conceding the argument already. Many of us would happily accept these sorts of limitations on a wealth tax if it means there is a wealth tax.
Predatory property taxes were a part of how so many black American farmers lost their land: https://archive.is/exkhR