While this is indeed a problem with alignment, we are essentially at the level of a cargo-cult when it comes to getting AI to be aligned with literally any values, including the values of the corporation who ran their training:

We're copying morality and instruction following that seems to work on humans without really understanding why it seems to work on humans, and grading outputs much as if the outputs came from a human.