alt Hacker NewsThere's still the question of access to the codebase. By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt. The attacker usually has even less access than this - in the beginning, they have network tools, an undocumented API, and maybe some binaries.
You can do a lot better efficiency-wise if you control the source end-to-end though - you already group logically related changes into PRs, so you can save on scanning by asking the LLM to only look over the files you've changed. If you're touching security-relevant code, you can ask it for more per-file effort than the attacker might put into their own scanning. You can even do the big bulk scans an attacker might on a fixed schedule - each attacker has to run their own scan while you only need to run your one scan to find everything they would have. There's a massive cost asymmetry between the "hardening" phase for the defender and the "discovering exploits" phase for the attacker.
Exploitability also isn't binary: even if the attacker is better-resourced than you, they need to find a whole chain of exploits in your system, while you only need to break the weakest link in that chain.
If you boil security down to just a contest of who can burn more tokens, defenders get efficiency advantages only the best-resourced attackers can overcome. On net, public access to mythos-tier models will make software more secure.
Replies
On that latest episode of 'Security Cryptography Whatever' [0] they mention that the time spent on improving the harness (at the moment) end up being outperformed by the strategy of "wait for the next model". I doubt that will continue, but it broke my intuition about how to improve them
[0] https://securitycryptographywhatever.com/2026/03/25/ai-bug-f...
The problem, though, is that this turns "one of our developers was hit by a supply chain attack that never hit prod, we wiped their computer and rotated keys, and it's not like we're a big target for the attacker to make much use of anything they exfiltrated..." into "now our entire source code has been exfiltrated and, even with rudimentary line-by-line scanning, will be automatically audited for privilege escalation opportunities within hours."
Taken to an extreme, the end result is a dark forest. I don't like what that means for entrepreneurship generally.
Most companies and their vendor ecosystems run on OSS
Worse, "attackers no longer break in, they log in", so the supply chain attacks harvesting credentials have been frightening
Still it makes cost of making software higher.
You cannot get away with „well no one is going to spend time writing custom exploit to get us” or „just be faster than slowest running away from the bear”.
> By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt
Primitive? I'd say simple and thorough.
One defender, many attackers, I don't see how the economy of scale can be positive for the defender.
Assuming your code is inaccessible isn't good for security. All security reviews are done assuming code source is available. If you don't provide the source, you'll never score high in the review.
> There's a massive cost asymmetry between the "hardening" phase for the defender and the "discovering exploits" phase for the attacker.
Well, you need to harden everything, the attacker only needs to find one or at most a handful of exploits.
> By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase
What accounts are these?
I've seen some people use this but I cannot imaging that anyone thinks this is the best.
For example I've had success telling LLMs to scan from application entry points and trace execution, and that seems an extremely obvious thing to do. I can't imagine others in the field don't have much better approaches.
>By all accounts, the best LLM cyber scanning approaches are really primitive
It seems like that is perhaps not the case anymore with the Mythos model?
what about open source software?
[dead]
> it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt.
This really is not the case.
You have freedom of methodology.
You can also ask it to enumerate various risks and find proof of existence for each of them.
Certainly our LLM audits are not just a prompt per file - so I have a hard time believing that best in class tools would do this.