Six curl CVEs after OpenAI and Anthropic came back with zero

Wow, this announcement is good content marketing.

Don't get me wrong, it's interesting. But there is no technical discussion as to how they did it. It's simply: we did it and Mythos and Codex didn't.

It's good to know that it's possible, but I'd have already expected it. Put a base model versus a base model + harness + whatever else, and yea, if you do it right then you have a better system to find vulnerabilities.

> We then ran AISLE's autonomous AI system against curl.

They don't even mention what models the use under the hood. It wouldn't surprise me if they are from Anthropic and OpenAI.

We had a few AISLE-generated security reports, and the signal to noise was reasonably good.

The most notable bug/exploit their scanner found was: https://gitlab.com/nbdkit/libnbd/-/commit/e50bbd2681117c2dd8...

The tool basically had to chain two exploits together to reach this. It also came up with a patch to fix which was fairly sensible (but I ended up editing it further for clarity).

Since AISLE reported 29 issues but only 6 warranted a CVE, and all the found CVEs were "low" severity, this makes me wonder if AISLE simply is tuned for a higher false positive rate than the anthropic and openai tools (which may have found the same 6 issues and decided not to report them)

Curl seems to becoming one of the favourite things to demo AI finding vulns.

Curl is going to end up incredibly secure.

Very unrelated to the content of the article, but that is a pretty weird ft ligature in the heading. It looks a letter from another alphabet. Which maybe makes this pretty cool after all.

That's bragging rights correctly earned, i think! As marketing-y as this post is, definitely something to keep an eye on.

I also have some secret recipe finding one class of bugs: https://github.com/tmux/tmux/issues?q=is%3Apr%20author%3Ablm...

curious they're willing to run AISLE on tmux to find more than mine.

Good marketing and definitive proof that local (read: on-prem & air-gapped) models with correct context and tools are good enough to perform on par and above SOTA cloud hosted solutions.

We have seen this point many times before with different technologies. The first computers at university were big and expensive, same as this machine. Give it a few years and this functionality will be a commodity.

I like the looks of Aisle and what they stand for...

That being said you cannot compare a model with a specialised harness. These are two completely different things.

Am I missing something?

Given enough AIballs all bugs are shallow

The gauntlet has been thrown. Will Anthropic or OpenAI pick it up?

One does not "discover" a CVE like this. To discover a CVE would mean you searched for a particular piece of software and found it vulnerable according to the NVD. That's not a novel discovery by any means.

What they did is they found bugs and that they were exploitable in certain edge cases. As the bugs turned out to be vulnerabilities, they were assigned a CVE in the NVD with low severity.

IMHO Aisle stockedpiled too much in the marketing shelves.

OpenAI and Anthropic have both been studying CURL for a while though. Anything they found was already fixed.

If you want to compare you need to start with something that none of studied. Somebody please take the source to a 2023 release of CURL (It shouldn't be hard to find one) - before all the current AI craze, and run all the tools on them to see what they find. Only then can we compare numbers. (and even then severity may come into place - all 6 are rated low impact)

"your LLM is cool, but can it find vulns in curl"

Marketing Slop

This is an ad. I didn't learn anything from reading it.