Tuning the model so far in the direction of being aggressively useful that it will quickly go off the rails in the name of helpfulness.

I swear I spend more time telling Claude not to do things than telling it what to do.

Others have given examples, but here's the theory: https://www.lesswrong.com/posts/fuSaKr6t6Zuh6GKaQ/when-is-go...

Reinforcement Learning (in LLMs) trains via gradient descent on a reward signal that's an imperfect proxy for the actual goal of the engineers doing the training. So, under mild optimization pressure, you get increasingly more of what you want, because that's the easiest way to increase the metric.

But as the optimization pressure increases, so do the ways to increase the metric by doing increasingly weird things. If the full action space grows sufficiently faster than the "things you actually want" subset, the amount of "things you actually want" goes to 0 under sufficient RL.

In this context, benchmaxing, if you will, so hard towards agentic coding benchmarks that everything else suffers.

See 5.6, Astra, and Opus 4.8 for examples