Sometimes I ssh into a server as a specific user (e.g. as the "app" user that is used to run a web app), sometimes only root is available (probably not best practice, but it's not like I can or want to fix it myself).

In any case it's not practical to carry your dotfiles everywhere you go. Changes are also a hassle to propagate