I believe I got rid of most of the issues with some .ssh setting which simply uses the normal xterm I believe. I can't check right at this moment, but apparently, simply using xterm and not the xterm-xkitty or what it's called does not actually make much problems when ssh-ing into a machine - not in my case at least.
It was a setting in some ~/.ssh config file though I believe, not a kitty setting.
I currently just use this in my .zshrc:
alias ssh="kitty +kitten ssh"
sshpass() { TERM=xterm-256color command sshpass "$@"; }
Could also alias sshpass differently but I mostly use that for embedded devices where I just want a simple shell anyways.
The alternative you mean is in ~/.ssh/config:
Host *