alt Hacker NewsThe PHP2Shell vulnerability ended up being paradoxically comforting for me.
They turned frontier models loose on PHP, and the big one they found was based on a previously known logic flaw that was extremely difficult to exploit. Which makes sense: hence the need for frontier models.
And in the wild, while there have been many attempts and a few simple demo successes, the practical impact has seemed to be small, seemingly constrained by the length of the chain and the diversity of PHP configs.
In short, it was indeed a major vulnerability, but not magic (in the Arthur C Clarke sense).
I think ultimately the fear comes down to: do you believe that code can be secured. I recognize that there is probably no objective way to deductively prove it because of the Halting Problem etc. But if one believes code can be constructed to operate to spec, then there must be a limit to the damage an LLM can do, even a very smart one.
I think we are going through a phase where a lot of known triaged code shortcomings will get exposed and hopefully fixed. Sucks while it is happening. But not actually the first round of this type of evolution. And ultimately I think as before, we will come out the other side with more secure and durable technology.
Replies
The halting problem is a product of a Turing-complete model of computation.
If we want perfectly secure systems, then perhaps we don’t want universal and general purpose ones.
I think the more fundamental question than "can code be secured" is something along the lines of "can we properly measure the security of the various components of a sufficiently codebase". Security consultants have workflows but ultimately everything is ad-hoc and suffers from cominatorial explosion, and what's missing is the security equivalent of Lean, a real formalized effort to prove security. There have been many attempts, of course. One of the great problems of our era.
Thus far, these agent vulnerabilities are all demonstrating million-monkey intelligence, rather than "super" intelligence, in the sense of some swarm of robo-Neo super hackers.
That's consistent with my experience using them for coding, where even the best models mostly work by being persistently midwit (with infinite knowledge) more than individually brilliant.
One corollary of this is probably that any project that has had an army of actual monkeys pounding on it for years is likely more resistant to LLM attack than one that has not. Everyone is worrying about individual robot geniuses finding obscure vulnerabilities in Linux, PHP, Python, etc., but they should probably be more worried about the permutations of less-known software floating around their systems.
The expectations of AI models in security really do seem to be magical in a way that, I think, obscures what needs to be done. Of course AI can’t magically create exploits that somehow drill through otherwise secure systems, because computers don’t work that way. It’s just a matter of not taking for granted things that worked in the past, like downloading code from the Internet and running it in a super-duper secure sandbox.
A computer that isn’t connected to anything (make sure to disable Bluetooth and the microphone) is already safe. I bet eventually we’ll be able to serve static HTML pages safely eventually. Maybe not images. Font selections might have to be pretty restricted. But, at least we’ll have the useful part of the internet back.
> and hoppefully fixed.
Well, the report you're replying to doesn't make me so hopeful: "7349 findings made public, 217 (.5%) fixed"
> I recognize that there is probably no objective way to deductively prove it because of the Halting Problem etc.
There is. But it will only be reliable if every transitive dependency of your codebase, all the way down to the hardware level, has been formally verified. Failing that, you have to rely on opaque contracts at the boundaries, and those are just promises that may be mistaken.
The Halting problem, Rice's theorem, etc. only apply if your goal is to automatically and completely deduce properties or their violations for arbitrary programs. If your goal is simply to prove security and correctness properties for well-understood classes of programs in industry, then there are no fundamental obstructions.