logoalt Hacker News

ajrossyesterday at 6:53 PM0 repliesview on HN

This is hilarious, but it's also not crazy surprising? It's an example of a "hidden context" question that we see all the time on exams that trip all of us up at one time or another. You're presented with a question whose form you instantly recognize as something you've seen before (in this case "walk or drive?"), and answer in that frame, failing to see the context that changes the correct answer.

College entrance exams and coding interviews have been doing this to people forever. It's an extremely human kind of mistake.

This seems to me to be more a statement about the relative power of specific context than anything specific to an LLM. Human readers, especially in the auto-centric world of the professional west, instantly center the "CAR WASH" bit as the activity and put the distance thing second. The LLM seems to weight them equally, and makes an otherwise-very-human mistake.

But ask someone who doesn't own a car? Not sure it's as obvious a question as you'd think.