alt Hacker NewsThe thing with me and this is that the teams that were competing in the DARPA Grand Cyber Competition all had this capability, like, last year.
All the attention has been on software security, of extracting the next marginal vulnerability out of heavily-scrutinized large codebases. In the actual professional field of infosec, that's a speciality; another specialty is network pentests and red-teaming, which exploits misconfigurations and seeks out weakest-link software (rather than exhaustively fishing for the next kernel LPE or whatever).
Red teaming and netpen work is probably substantially easier for models than software security; it costs less context, but is also much more explicitly an implicit search problem where win conditions are just spotting stupid stuff that humans missed.
My visceral reaction to this is that with the right harness, you probably could have replicated this with an open-weights model last year. (I'm saying this as confidently as I am because a CGC team leader agreed with me about it yesterday).
I think people forget that the harness work we're considering here --- I don't know anything about OpenAI's harness or ExploitGym or whatever --- are basically not new; people have been developing automated exploitation and pivoting toolkits for decades, and scanners long before that. So the idea of a tool getting 0.0.0.0/0 as a target list instead of 192.168.1.0/24 and then busting up a bunch of random people's computers: not really very startling.
Obviously, LLMs give those kinds of scanners an intentionality they wouldn't have had before. But as a person who keeps a computer science perspective on security stuff, I don't know that it gives them capabilities they didn't have.
Replies
Isn’t the intentionality the actually concerning bit? Exploit capabilities are all fun and games constrained by the humans directing them; a paperclip maximizer going rogue with them is less fun.
I still think there’s something to be said here for generality. This does not appear to been designed as a cyber pen test tool with specialized harness. From what I understand they were testing GPT-6 in an agent system with GPT-5.6 subagents. It me it’s amazing that a general model could excel on a huge range of tasks like this and new capabilities emerge when a model is multidisciplinary and can combine knowledge and skills from many separate domains.
re: your open models can do this point, I immediately wondered after reading about this, if OpenAI publicizing this so much is just more potential rationale to try to get US GOVT to ban Chinese open models.. (eg as in, "if gpt/fable can do this then K3 probably can too, but WE are the safe US models with guardrails and who knows what the evil Chinese models might allow," etc)
Echoing other responses to you but this isn't a capability problem. You're totally right that this is so last year in terms of LLMs being capable in infosec. The issue here is an alignment one, i.e. the model seemingly isn't "aware" (especially with its guardrails turned off it would seem) that it is doing something immoral/illegal by hacking HF for the answer to its (vague) query of "solve this problem". Or if it is "aware", it's not trained to care, i.e. it's not an aligned model (alignment is considered hard).