You can but with agents you might want a security boundary/faux-airgap between the agent and your machine. I don't think you should trust X11/Wayland forwarding to maintain strong isolation against a malicious server in that scenario.
TUIs are superior in that case because they cuts the attack surface down to a regular SSH session.
Agreed. I sandbox agents using containers which also allows me to inject credentials and all sorts of things.
I think there's 2 clear layers in these coding tools: the runtime engine that lives ON the system, can execute commands, read/write files, make network calls and the UI, which can be a local (TUI) or remote.
A remote UI allows you to manage multiple remote runtimes concurrently and can even be on your phone.
Anyone preferring Desktop vs TUI is not managing multiple systems I would guess.