You don't even need there to be any kernel of reality statistically.

If a model is trained on data that shows some bias due to prior human decisions, it would continue to follow those biases while being a lot harder to prove/sue. There were examples of this with AI assisted resume screening & criminal sentencing for example.