saw an interesting hypothesis about this ( quoted from Jake Mendel at https://www.lesswrong.com/posts/eKe7fky8kSt9DCJxZ/openai-ast... ):

"It's probably that in capabilities RL the models are rewarded however they complete the task. Whereas in RL related to alignment or safety, they're more likely to be being tested for their ethical judgment or rule adherence (ie theyre supposed to be a good nice well-behaved AI rather than a ruthless task completion AI). So they're more likely to refuse or have some ethical issue or stop and ask for help a suspicious amount of the time if the task looks more like alignment RL environments than capability RL environments."