logoalt Hacker News

rybosometoday at 4:41 PM1 replyview on HN

Very much agreed on the significance.

The lack of a true airgap should have been identified as a critical weakness and addressed with not only additional layers trying to prevent escape, but at minimum an alarm which would page a human when escape did occur.

My guess is this occurred in a setting where, to be frank, there were too many researchers and not enough software engineers and SREs.

All of the systems which were initially built largely or exclusively by researchers - inference, evaluation, training - are at the level of complexity and significance that they need systems experts. Maybe some teams don’t have access to them. I know plenty of software engineers are employed at OAI but I’d wager they’re concentrated in inference and training, rather than evaluation?

The ironic part is if you had presented this setup to chatGPT and asked how to improve it and if it was good enough, you’d have gotten a ton of actionable suggestions which would have mitigated or prevented this.


Replies

mnickytoday at 5:11 PM

The air gap would probably help and after this incident I hope labs will think about using such a measure when appropriate.

On the other hand I think that proper solution for these kinds of problems is not at a sandbox level, but at a model alignment level.

Also it shows that maybe the most serious risk comes not from releasing models publicly but from internal, pre-release period where you sometimes need/want to lift some guardrails a bit etc.