alt Hacker NewsThat's why this paper is important - it shows it isn't trained out. Leaving no other information in the model makes it clear what the biases are, and that the model is willing to make a biased decision. If you give it other unbiased criteria as well the bias may still easily remain but not be as clear.
Not sure it’s that strong. The prompt gives the presumption that this matters. Not necessarily a training issue vs the prompts being poorly written and the results being inherent in the bias they carry