> If the TUI hijacks the mouse and does something different with it (e.g. vim switching into visual mode) that is just annoying.

tmux is great for this. Just press Ctrl+[ and the cursor can select whatever text is in the window.