You don't need a private key on the host for that, only your public key in authorized_keys.

Edit: Oh boy I should have paid more attention. Those are the host keys. :facepalm: