My experience is that Emacs wants to be your everything, and works best if you let it. Have you tried putting your terminals inside of Emacs instead of the other way around?

* If you need a "real" terminal emulator, you can use something like vterm (https://github.com/akermu/emacs-libvterm/).

* If you need to be able to attach/detach Emacs sessions on remote machines, you can use something like dtach or abducto (https://www.brain-dump.org/projects/abduco/).