I don't mean this flippantly, but it's usually whoever you're writing the check to. My first home had the mortgage sold within a week, and someone else started taking the money out of my account.
Oh okay, in that case mine were never sold. As in the sibling reply, my assumption is that the "retail" side of the mortgage (who you make your payments to) is fully detached from who ends up owning the mortgage backed securities that your debt is being financed by.
The servicing company could be different than whoever actually owns the debt.