Versions are different during the upgrade. That's how upgrade is done.

You spin up replica, upgrade it to the target engine version, ensure it's up to date and switch over. [1]

[1] https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...