Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing. I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned. When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this. And then the model did the same thing few days later since it was taught to do that.

I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.