I regretfully suspect this actually is the future of UI. You just need smart local models running fast enough to keep the framerate high, but that’s inevitable in the long term, isn’t it?
What if we're really just living in a world model? How big of a GPU is required to run the universe I wonder?
Why would you do that instead of having the model do the task you're using the UI for? It could generate UI widgets on-the-fly as needed to take richer inputs.
We had decades to work on making software more performant, but instead most users' space-age 2nm chips spend most of their time running Javascript because it was faster to make software that way.