IRC is also most commonly used for open servers where anyone can join whenever they want to without as much as needing to register for an 'account'! You just pick a nickname out of thin air and off you go.
In that kind of environment, end to end encryption really doesn't add value.
The IRC admins can read all your messages, be it to a channel or to another user.
Even without registering my nick, I would expect a modern protocol to keep my pm communication private by default.