alt Hacker NewsRight, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?"
You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario generates stratified output. That's it.
It's both almost certainly true that Fable 5.1 mimblewort vs. bafflewick would show stratification, and that has ~no relevance on whether Fable is useful for software engineering work.
That's the point - those two made-up things should have resulted in an equal split.
That's just your claim about how LLMs "should" work, based on ... your subjective preference?
> That's just your claim about how LLMs "should" work, based on ... your subjective preference?
Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias.
Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?