That is how terminals work in OSes that were born with framebuffers instead of teletypes.

It is just another graphical application window on the OS.

someone made an x server that renders to sixel[0]

[0]https://news.ycombinator.com/item?id=45341683

I have been thinking about this for a while. It's not as crazy as it may sound, especially in light of the other comments making a parallel between terminals and notebooks.

A few thoughts:

1. Linux VTs kind of have this feature already: there is the normal buffer, the alternate buffer (that something like htop would draw on), and an IOctl can change them to/from graphics mode.

2. It makes sense for interactivity. Kitty's graphics protocol is quite useful for static shapes, can be abused for animations, but doesn't really cut it for interactivity (say, pan a graph around). Wayland is designed for this.

3. Wayland would be a good fit: isolate each command from another, let them request buffers, but keep control of where to display them, do not update them when off screen, etc.

4. One downside is that terminals excel for one-shot tasks. What's the purpose of the display when you are done with it? Should you kill the process driving it? Due to this, it may make more sense to delegate more features to the terminal emulator (displaying the 3D model, etc). Or maybe just allow the app to temporarily take over the window.

5. Once you have it up and running, have it talk directly to the direct rendering manager. Your "kmscon" is now your compositor / desktop environment. That's a fun thought! Add some basic terminal features like tabs and tiling, and you've inverted the usual setup.

6. One downside is accessibility. I really like that I can copy-paste any part of the interface for reference, "screenshots", etc. It's good for screen readers, too. You lose these advantages by going to Wayland.

7. Another current terminal limitation is fonts. Power line, yazi & other make use of custom fonts for drawing part of the interface, logos, etc. AFAIK there is no good way to query their availability (which is also an issue for color emoji). Custom fonts or a new protocol could be useful, but client apps could draw it themselves if given a surface (that can already do that with the kitty graphics protocol, mind you)

Obviously I am not seriously considering to make such a terminal emulator, but it would be an interesting experiment (heck, maybe something I should try this "vibe coding" with, since I wouldn't want to spend too much time on it).