alt Hacker NewsI think your 3k figure comes from here - It is explained:
> As judges, the professors then completed 2,918 blinded, forced-choice comparisons (median per judge: 200), each time indicating which of the two anonymized responses, from the instructor or the LLM, they would rather give to a student
So did were the answers fact checked? If not that seems like a pretty obvious flaw!