I think I mostly just want to see someone do it so I can see what it looks like; it just feels like something that should exist. You're taking baby steps towards it! :)
I've been working on one version of this, if you want to see what it looks like: https://bateri.dev/pens
It's narrower than replacing the cell grid. A program asks for a view the way vim takes the alternate screen. If the terminal answers, the program hands it the view's files once, and from then on the two exchange six small messages inside the normal byte stream (APC frames, the same trick kitty graphics uses). No port and no extra connection, so it works over plain ssh to a box with no GPU. When the program exits you get your terminal back, and in a terminal that doesn't answer the program can fall back to plain text.
A view is either a Lua script the terminal draws itself, in its own font and theme, or a sandboxed web page. The second one is exactly the "terminals are becoming browsers" worry from upthread, and I don't think that worry is wrong, which is why the native kind exists. Going through tmux is a non-goal: a multiplexer that doesn't pass the stream through untouched just means the program sees no host and falls back.
The only host so far is bateri, the macOS terminal I'm building, and the spec is an early Apache-2.0 draft: https://github.com/bateri/pencere/blob/main/spec/draft/pence...
I kinda keep talking about it in the hopes that someone will be nerd sniped enough to throw tokens at it as an idea. Sort of what you're probably doing too I guess. ;)