As someone who often fails to read subtext, I would estimate that most people expect you to participate in mind reading as a natural part of conversation.

So it is no surprise that many people have difficulty switching gears to literal mode when interacting with these models.