alt Hacker NewsThere's Most Forbiddenish and there's Most Forbiddener. We already apply length penalties to CoT in RL, for instance.
Sometimes I worry that the framing around CoT grading prevents research into necessary and useful safety techniques that edge up against it, but what do I know I'm just a guy, c'mon n downvote me LW bros
Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?