> I see people building 6k+ machines to run the best of them at the moment, which are behind SOTA by maybe 6 - 12 months or so right now

SOTA in open source (frontier Kimi MOE) requires terabytes of RAM. At DDR5 prices, that's $40k alone. For HBM, higher. We're years away from consumer hardware matching the power and latency of e.g. Claude.