alt Hacker NewsCybersecurity looks like proof of work now
Comments
The article heavily quotes the "AI Security Institute" as a third-party analysis. It was the first I heard of them, so I looked up their about page, and it appears to be primarily people from the AI industry (former Deepmind/OpenAI staff, etc.), with no folks from the security industry mentioned. So while the security landscape is clearly evolving (cf. also Big Sleep and Project Zero), the conclusion of "to harden a system we need to spend more tokens" sounds like yet more AI boosting from a different angle. It raises the question of why no other alternatives (like formal verification) are mentioned in the article or the AISI report.
I wouldn't be surprised if NVIDIA picked up this talking point to sell more GPUs.
Relevant Tony Hoare quote: “There are two approaches to software design: make it so simple there are obviously no deficiencies, or make it so complex there are no obvious deficiencies”.
Security has always been a game of just how much money your adversary is willing to commit. The conclusions drawn in lots of these articles are just already well understood systems design concepts, but for some reason people are acting like they are novel or that LLMs have changed anything besides the price.
For example from this article:
> Karpathy: Classical software engineering would have you believe that dependencies are good (we’re building pyramids from bricks), but imo this has to be re-evaluated, and it’s why I’ve been so growingly averse to them, preferring to use LLMs to “yoink” functionality when it’s simple enough and possible.
Anyone who's heard of "leftpad" or is a Go programmer ("A little copying is better than a little dependency" is literally a "Go Proverb") knows this.
Another recent set of posts to HN had a company close-sourcing their code for security, but "security through obscurity" has been a well understand fallacy in open source circles for decades.
I mostly agree with the article.
> You don’t get points for being clever
Not sure about this framing, this can easily lead to the wrong conclusions. There is an arms race, yes, and defenders are going to need to spend a lot of GPU hours as a result. But it seems self-evident that the fundamentals of cybersecurity still matter a lot, and you still win by being clever. For the foreseeable future, security posture is still going to be a reflection of human systems. Human systems that are under enormous stress, but are still fundamentally human. You win by getting your security culture in order to produce (and continually reproduce) the most resilient defense that masters both the craft and the human element, not just by abandoning human systems in favor of brute forcing security problems away as your only strategy.
Indeed, domains that are truly security critical will acquire this organizational discipline (what's required is the same type of discipline that the nuclear industry acquires after a meltdown, or that the aviation industry acquires after plane crashes), but it will be a bumpy ride.
This article from exactly 1 year ago is almost prophetic to exactly what's going on right now and the subtle ways in which people are most likely to misunderstand the situation: https://knightcolumbia.org/content/ai-as-normal-technology
> to harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them.
I, for the NFL front offices, created a script that exposed an API to fully automate Ticketmaster through the front end so that the NFL could post tickets on all secondary markets and dynamic price the tickets so if rain on a Sunday was expected they could charge less. Ticketmaster was slow to develop an API. Ticketmaster couldn't provide us permission without first developing the API first for legal reasons but told me they would do their best to stop me.
They switched over to PerimeterX which took me 3 days to get past.
Last week someone posted an article here about ChatGPT using Cloudflare Turnstile. [0] First, the article made some mistakes how it works. Second, I used the [AI company product] and the Chrome DevTools Protocol (CDP) to completely rewrite all the scripts intercepting them before they were evaluated -- the same way I was able to figure out PerimeterX in 3 days -- and then recursively solve controlling all the finger printing so that it controls the profile. Then it created an API proxy to expose ChatGPT for free. It required some coaching about the technique but it did most of the work in 3 hours.
These companies are spending 10s of millions of dollars on these products and considering what OpenAI is boasting about security, they are worthless.
It looks like proof of work because:
> Worryingly, none of the models given a 100M budget showed signs of diminishing returns. “Models continue making progress with increased token budgets across the token budgets tested,” AISI notes.
So, the author infers a durable direct correlation between token spend and attack success. Thus you will need to spend more tokens than your attackers to find your vulnerabilities first.
However it is worth noting that this study was of a 32-step network intrusion, which only one model (Mythos) even was able to complete at all. That’s an incredibly complex task. Is the same true for pointing Mythos at a relatively simple single code library? My intuition is that there is probably a point of diminishing returns, which is closer for simpler tasks.
In this world, popular open source projects will probably see higher aggregate token spend by both defenders and attackers. And thus they might approach the point of diminishing returns faster. If there is one.
I don't understand the nature of the supposed security incidents found by LLMs:
Are these totally previously unknown security holes or are they still generally within the umbrella of our understanding of cybersecurity itself?
If it's the latter, why can't we systematically find and fix them ourselves?
Then the question: what is cheaper, secure a code base written by humans, or secure a code base vibe coded with an army of agents?
The cost of this is going to come down dramatically - just throwing the model at the codebase is a really inefficient process. My own experiments show that spending more tokens on understanding and transforming how the codebase can be explored(i.e enumerating source to sink traces) drastically lowers the cost to confirm vulnerabilities.Something that excites me greatly is that software quality has been incredibly difficult primarily because no single developer can hold the entire contract in their head and analyze it. It's now a reality that we can transform raw source code into actionable artifacts that allow a system to see the big picture and pin point the fracture points within it.
"Security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them. To harden a system you need to spend more tokens discovering exploits than attackers will spend exploiting them." - This feels similar to missile defense dilemma. Spending 2M$ missile to attack a 20k$ drone.
I discussed this in more detail in one of my earlier comments, but I think the article commits a category error. In commercial settings, most of day-to-day infosec work (or spending) has very little to do with looking for vulnerabilities in code.
In fact, security programs built on the idea that you can find and patch every security hole in your codebase were basically busted long before LLMs.
By using these services, you're also exfiltrating your entire codebase to them, so you have to continuously use the best cyber capabilities providers offer in case a data breach allows somebody to obtain your codebase and an attacker uses a better vulnerability detector than what you were using.
Long ago, during the Viet Nam conflict, the US government learned that computers needed to be able to securely process data from multiple levels of classification simultaneously. Research in the 1970s found solutions that were adopted in the Mainframe world, like KeyKOS and EROS. Then the PC revolution swept all that away, and we're here 40+ years later, with operating systems that trust every bit of code the user runs with that user's full authority.
It's nuts. If the timing were slightly different, none of this "Cybersecurity" would even be a thing. We'd just have capabilities based, secure general purpose computation.
It looks like it, but it isn't. It's the work itself that's valued in software security, not the amount of it you managed to do. The economics are fundamentally different.
Put more simply: to keep your system secure, you need to be fixing vulnerabilities faster than they're being discovered. The token count is irrelevant.
Moreover: this shift is happening because the automated work is outpacing humans for the same outcome. If you could get the same results by hand, they'd count! A sev:crit is a sev:crit is a sev:crit.
Give more ammo to bad actors and sell the ammo to defenders, charge both for tokens. Why isn't this business model banned already?
I've said for decades that, in principle, cybersecurity is advantage defender. The defender has to leave a hole. The attackers have to find it. We just live in a world with so many holes that dedicated attackers rarely end up bottlenecked on finding holes, so in practice it ends up advantage attacker.
There is at least a possibility that a code base can be secured by a (practically) finite number of tokens until there is no more holes in it, for reasonable amounts of money.
This also reminds me of what I wrote here: https://jerf.org/iri/post/2026/what_value_code_in_ai_era/ There's still value in code tested by the real world, and in an era of "free code" that may become even more true than it is now, rather than the initially-intuitive less valuable. There is no amount of testing you can do that will be equivalent to being in the real world, AI-empowered attackers and all.
What do they mean when they say "no diminishing returns?" does this essentially mean the code you are testing has no bounded state space and you continue to find infinite paths?
Because we have tools and techniques that can guarantee the absence of certain behavior in a bounded state space using formal methods (even unbounded at times)
Sure, it's hard to formally verify everything but if you are dealing with something extremely critical why not design it in a way that you can formally verify it?
But yeah, the easy button is keep throwing more tokens till you money runs out of money
As a result of all this AI "find a zero-day" business, when I boot to windows I open the task manager and order by pid. I kill anything I didn't start or don't recognise.
The only process that scared me was windowgrid. It kept finding a way back when I killed all the "start with boot" locations I know. Run, runonce, start up apps, etc. Surely it's not in autoexec.bat :)
I'm curious to see if formally verified software will get more popular. I'm somewhat doubtful, since getting programmers to learn formally math is hard (rightfully so, but still sad). But, if LLMs could take over the drudgery of writing proofs in a lot of the cases, there might be something there.
> Cybersecurity looks like proof of work now
Imo, cybersecurity looks like formally verified systems now.
You can't spend more tokens to find vulnerabilities if there are no vulnerabilities.
really, really?
After how many years of "shifting left" and understanding the importance of having security involved in the dev and planning process, now the recommendation is to vibe code with human intuition, review then spend a million tokens to "harden"?
I understand that isn't the point of the article and the article does make sense in its other parts. But that last paragraph leaves me scratching my head wondering if the author understands infosec at all?
To me it looks like formal verification is going to be the answer. We're going to move up the ladder and write formal specs and proofs soon.
Trusted software will be so expensive that it will effectively kill startups for infrastructure, unless they can prove they spent millions of dollars hardening their software.
I predict the software ecosystem will change in two folds: internal software behind a firewall will become ever cheaper, but anything external facing will become exponential more expensive due to hacking concern.
Maybe I’m missing something, but there’s also the idea that you don’t need to be perfectly secure, you just need to be secure enough that it’s not worth the effort to break in.
In the case of crooks (rather than spooks) that often means your security has to be as good as your peers, because crooks will spend their time going with the best gain/effort ratio.
> If corporations that rely on OSS libraries spend to secure them with tokens, it’s likely going to be more secure than your budget allows.
That's a really big "if". Particularly since so many companies don't even know all of the OSS they are using, and they often use OSS to offload the cost of maintaining it themselves.
My hope is when the dust settles, we see more OSS SAST tools that are much better at detecting vulnerabilities. And even better if they can recommend fixes. OSS developers don't care about a 20 point chained attack across a company network, they just want to secure their one app. And if that app is hardened, perhaps that's the one link of the chain the attackers can't get past.
Maybe code quality shouldn't be considered cybersecurity in the first place?
When things are tagged "cybersecurity", compliance/budget/manager/dashboard/education/certification are the usual response...
I don't think it would be an appropriate response for code quality issues, and it would likely escape the hands of the very people who can fix code quality issues, ie. developers.
There are never ending ways to make agents better at hacking. Defense is clearly behind. At my startup we are constantly coming up with new defensive measures to put our hacking agent Sable against, and I've determined that you basically need to be air gapped in the future for a chance of survival. A SOC of AI agents can't keep up with 1 AI hacker on a network that is even remotely stealthy. it is a disaster. wrote an article about it: https://blog.vulnetic.ai/evading-an-ai-soc-with-sable-from-v...
> You don’t get points for being clever. You win by paying more.
And yet... Wireguard was written by one guy while OpenVPN is written by a big team. One code base is orders of magnitude bigger than the other. Which should I bet LLMs will find more cybersecurity problems with? My vote is on OpenVPN despite it being the less clever and "more money thrown at" solution.
So yes, I do think you get points for being clever, assuming you are competent. If you are clever enough to build a solution that's much smaller/simpler than your competition, you can also get away with spending less on cybersecurity audits (be they LLM tokens or not).
The problem with the security researcher industry is that it is infested with self promoters who talk about methodologies and tools but have never written any secure software themselves. Or any software at all, as the GitHub accounts from some of these geniuses show.
Of course those are attracted to new tools and AI shill institutes like AISI (yes, the UK government is shilling for AI, it understands a proper grift that benefits the elites).
Security "research" is perfect for talkers and people who produce powerpoint graphs that sell their latest tools.
You still can sit down and write secure software, while the "researchers" focus on the same three soft targets (sudo, curl, ffmpeg) over an over again and get $100,000 in tokens and salaries for a bug in a protocol from the 1990s that no one uses. Imagine if this went to the authors instead.
But no, government money MUST go to the talkers and powerpointists. Always.
> Classical software engineering would have you believe that dependencies are good (we’re building pyramids from bricks)
Would it? I’m old school but I’ve never trusted these massive dependency chains.
That’s a nit.
We’re going to have to write more secure software, not just spend more.
The PoW analogy completely ignores the actual hard part: fixing the stuff. It’s cool if you burn millions of tokens to find 1,000 bugs, but it's completely useless if your small team only has the bandwidth to safely patch 5 of them without taking down prod.
If you have a limited budget of tokens as a defender, maybe the best thing to spend them on is not red teaming, but formalizing proofs of your code's security. Then the number of tokens required roughly scales with the amount and complexity of your code, instead of scaling with the number of tokens an attacker is willing to spend.
(It's true that formalization can still have bugs in the definition of "secure" and doesn't work for everything, which means defenders will still probably have to allocate some of their token budget to red teaming.)
>You don’t get points for being clever. You win by paying more.
Really depends how consistently the LLMs are putting new novel vulnerabilities back in your production code for the other LLMs to discover.
I don't know about Mythos but the chart understates the capability of the current frontier models. GPT and Claude models available today are capable of Web app exploits, C2, and persistence in well under 10M tokens if you build a good harness.
The benchmark might be a good apples-to-apples comparison but it is not showing capability in an absolute sense.
I'm starting to think that Opus and Mythos are the same model (or collection of models) whereas Mythos has better backend workflows than Opus 4.6. I have not used Mythos, but at work I have a 5 figure monthly token budget to find vulnerabilities in closed-source code. I'm interested in mythos and will use it when it's available, but for now I'm trying to reverse engineer how I can get the same output with Opus 4.6 and the answer to me is more tokens.
Cybersecurity has always been proof of work. Fuck, most of software development is proof of work by that logic. Thats why many attacks originate from countries were the cost of living is a fraction of the COL in the United States. They can throw more people at the problem because its cheaper to do so.
But I don't really get the hype, we can fix all the vulnerabilities in the world but people are still going to pick up parking-lot-USBs and enter their credentials into phishing sites.
All of the recent news read like something that could happen in a cyberpunk novel - AIs that defend vs AIs that do the attacks.
I think were are already here. I wrote something about this, if you are interested: https://go.cbk.ai/security-agents-need-a-thinner-harness
Although not an escape from the "who can spend the most on tokens" arms race, there is also the possibility to make reverse engineering and executable analysis more difficult. This increases the attacker's token spend if nothing else. I wonder if dev teams will take an interest.
Better to write good, high-quality, properly architected and tested software in the first place of course.
Edited for typo.
My first thought seeing the title: "always has been"
> This chart suggests an interesting security economy: to harden a system we need to spend more tokens discovering exploits than attackers spend exploiting them.
What's new?
It was always about spending more money on something.
Team has no capacity? Because the company doesn't invest in the team, doesn't expand it, doesn't focus on it.
We don't have enough experts? Because the company doesn't invest in the team, doesn't raise the salary bar to get new experts, it's not attractive to experts in other companies.
It was always about "spending tokens more than competitors", in every area of IT.
Security always had “defender’s dilemma” (an attacker needs to find one thing, but defender needs to fix everything) problem, nothing is new in terms of AI’s impact just application of different resources and units.
Everything in modern corporate is just proof of work. Security is filling out forms. Engineering is just endless talking. Token-maxing is the new meta.
If you run this long enough presumably it will find every exploit and you patch them all and run it again to find exploits in your patches until there simply... Are no exploits?
we did a lot of thinking around this topic. and distilled it into a new way to dynamically evaluate the security posture of an AI system (which can apply for any system for that matter). we wrote some thoughts on this here: https://fabraix.com/blog/adversarial-cost-to-exploit
I can see the dichotomy forming in the "post AI" world;
1) massive companies spending millions of tokens to write+secure their software
2) in the shadows, "elite" software contractors writing bespoke software to fulfill needs for those who can't afford the millions, or fix cracks in (1)
(Oh wait, I think this is what is happening now, anyway, minus the millions of tokens)
Does this mean all code written before Mythos is a liability?
There's still the question of access to the codebase. By all accounts, the best LLM cyber scanning approaches are really primitive - it's just a bash script that goes through every single file in the codebase and, for each one and runs a "find the vulns here" prompt. The attacker usually has even less access than this - in the beginning, they have network tools, an undocumented API, and maybe some binaries.
You can do a lot better efficiency-wise if you control the source end-to-end though - you already group logically related changes into PRs, so you can save on scanning by asking the LLM to only look over the files you've changed. If you're touching security-relevant code, you can ask it for more per-file effort than the attacker might put into their own scanning. You can even do the big bulk scans an attacker might on a fixed schedule - each attacker has to run their own scan while you only need to run your one scan to find everything they would have. There's a massive cost asymmetry between the "hardening" phase for the defender and the "discovering exploits" phase for the attacker.
Exploitability also isn't binary: even if the attacker is better-resourced than you, they need to find a whole chain of exploits in your system, while you only need to break the weakest link in that chain.
If you boil security down to just a contest of who can burn more tokens, defenders get efficiency advantages only the best-resourced attackers can overcome. On net, public access to mythos-tier models will make software more secure.