macOS should in general always extend for the first press, and then move relative to that decision. So if you select some text with the mouse and then shift-left/right it will extend the selection in whichever direction you chose. But after that first extension, you’re now in a “keyboard selection mode” where continuing to move in the first direction will continue to extend, but selecting the opposite direction will shrink the selection. And the up/down keys work the same and relative to the current extension direction (so if you were extending to the right, down will continue extending since going right at the end of the line would move you down, and up will shrink the selection by the same logic. And the reverse for extending to the left)

Mostly this makes sense to me, though it is admittedly annoying when you’ve selected a little too much with the mouse and want to deselect, you need to extend a bit first and then reverse direction.