As the gp says, only if you're logged in.

They've also been making it so the preference seemingly doesn't stick now. Used to be it would just randomly undo that setting, now it just doesn't respect it at all in my experience.