alt Hacker News> It's pretty easy for a reasonable person to determine when to call 911…
Unfortunately, it's fairly tough for someone to determine if they are themselves a reasonable person.
A model that only ever says yes is a cognitohazard, IMO. Like being a billionaire without the money to insulate you from the consequences.
Saying yes is not equal to "provides the requested information." If you use models for affirmation instead of information, this is a more reasonable concern.