Yep, most wealth is ultimately dependent on some kind of ability to scrape off tiny bits of practical accomplishments of value achieved by large numbers of people. In some cases, such as your housing example, this is a very direct dependency. In other cases the dependency is indirect and less visible.

The smarter ones among the rich have realized this and are open for something like a universal basic income precisely for that reason: to protect their wealth and the position in the food chain this wealth affords them.