Isn't this jaw-droppingly the wrong answer?! If the model isn't given any way to directly view the image shouldn't it just reply with one sentence asking for permission. This reads as a model hyper-trained to burn tokens.

Like suppose you issue a command to Opus or Fable which doesn't make sense and requires a lot of work. It will almost certainly not push back on your silly request and go ahead and burn as many tokens as it can doing the wrong task. This happens to me all the time.

Or if I was to tell a model to do something and it realized it could only do it by hacking into another service, by all means it should ask me if I want it to hack and not go ahead and do it by itself.

My jaw drops every day.

Small change in comparison, but today Fable created and benchmarked an RTree which was 100x faster to populate and 5x faster to query, compared to a previous attempt with Opus 4.6 a couple of months ago. That took about two coffees.

I think it's important to remember that as impressive Opus & co are, they're standing on the shoulders of giants, i.e. the engineers, academics and companies who have cooperated to design the incredible programming languages and machines we have today.

Others have pointed out enough issues such as ethical sourcing, but here’s 2 more:

1) It’s hard to be excited about something that has been promised to destroy your job.

2) New models come out bi-weekly with even more “amazing features”. Eventually you tune out because you can’t be at peak-amazement all the time. Tone down the language a bit…

Let’s say somehow the politics lined up and you were given huge hiring budget for your team. You went out and recruited what seemed like great talent. Weekly standups make it seem like everyone is ramped up and firing on all cylinders.

It’s great, right? But you look at what your team has actually delivered since—-and it’s just not that much more than last year.

I think that’s where a lot of us are at the micro and macro level. It seems really great from the inside and outside but we are still waiting to see the dramatic change in actual concrete software output.

Jacobian conjecture disproved? Doesn't even make it to the list.

> How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements?

Agree

> responded by writing its own computer vision pipeline

Was it "its own" or something that was part of its training material?

Don't get me wrong, I find this all amazing too and makes my work 10x easier and quicker. But it's not like it's inventing this stuff from scratch / first principles. It has seen this kind of tech before by consuming all publicly available source code and books etc. (And that's ok, but let's be honest/clear about that.)

I will be amazed when you can explain to me how it did that, i.e. in which weights exactly the knowledge about the CV pipeline was encoded, if there were any weights that were not contributing anything, how the planning worked, how it could scrape the list of subtasks it was currently working on from its context window, how it understood when to return to a higher-level task and which task, what kind of internal representation it derived from the pixel values, how it translated that representation into CAD code, etc etc.

Until then, it's just "hey, someone out there can do all those things and they're for rent for subsidized prices".

That's more frustrating than impressive.

While this is certainly impressive, I believe it is not a good answer from a user's perspective. Instead of having Claude automatically spend tokens in an unrelated task, I would have preferred it to first tell me: "I can't view the drawing. Would you like me to build a computer vision pipeline to do it"?

Maybe we’re desensitized because all that progress hasn’t made our lives any more meaningful, happier, or even easier.

ChatGPT released in 2022. We been promised to reach AGI in 2024, then 2025, then 2026, then 2028. It is 2026 and we still do not have AGI. In 2028 we will have jump to 40% in ARC-AGI 4, and you will say "Oh it's been less than 6 years since ChatGpt thing".

"However, in this task, the model was intentionally given no way to directly view the drawing." I consider this claim to be a rumor. Judging by the recent leak of the Claude CLI source code, such directives are hardcoded and sent with the system prompts. Furthermore, we don’t know what happens to your original prompts once they enter the API.

I'm tired boss

Because the step from earlier GPT models to this is not very large? Yes it's impressive they have forced it into learning that it has to get the job done no matter what, which usually means breaking implicit expectations and rules, but then again, Claude is not for people who care.

This isn't groundbreaking in any way. It's cool, but that's about it.

I asked Opus 4.8 to help me design an adapter for a 3d printed part and it actually modeled and GAVE ME the part.

That blew my mind. It doesn't surprise me that Opus 5 ups the ante.

I don't find it jaw-dropping because the idea is quite simple. Just feed the model an enormous amount of unethically sourced data, build data centers that cause droughts, and use all chips available so that normal people can't afford to buy RAM anymore.

We're all sacrificing great things in order to make these models more capable. Whether it'll all be worth it, only time will tell.

The problem is it should ask you before doing these things. This is a cherry picked example. There are times when they go on misadventures.

How does it do on this?: https://www.youtube.com/shorts/yN79mCiT5cU

I read this as the model being less steerable. I was bitten by this just today where I had a local Postgres instance running, and prompted opus 5 to run a server against it, but forgot to give it the password. Instead of asking for the password or mentioning anything, it “decided” that it should run the whole stack in a local Kind cluster to circumvent this. You can phantasize it all you want but this ultimately made my job harder than it needed to be and burnt a lot of unnecessary tokens. But hey, W for anthropic I guess.

edit: typo

> We should find a way to get "re-sensitized" to what we are witnessing and the pace of it.

We (collectively, there were obviously many exceptions) didn't internalise exponential growth even with the much faster doubling time of COVID-19 before the lockdowns hit. "Oh, it's just flu; wait, why is the supermarket short of hand sanitiser and bulk carbohydrates? Let's blame China and everyone who tells us to wear face masks!"

Same for the slower, but entirely foreseen, rate of climate change. "Who cares, it's just a few degrees, and anyway China's not going to cut emissions; wait why is the sky orange? Let's blame Canada and put tariffs on Chinese cars and PV!"

AI? "Who cares, it's just a stochastic parrot/glorified autocomplete. What's the Jacobin conjecture and why should I care, it's just brute-force."

Still, this is currently spiky intelligence, so I'm hoping some expensive-but-zero-to-few-fatalities catastrophic error forces better practices. An AI analog of the (1940) Tacoma Narrows Bridge (one canine fatality), rather than a repeat of Chernobyl or the (1984) Union Carbide incident in Bhopal (3,787-16k+ dead, ≥558,125 injured).

It's being said they were benchmaxxing.

> However, in this task, the model was intentionally given no way to directly view the drawing. Opus 5 responded by writing its own computer vision pipeline to pull the geometry from the raw pixels

Surely being able to view the raw pixels counts as viewing the drawing... How else does a computer view a drawing?

Computer vision != ML pipeline

Is this a bot? Sounds like ai

I mean I hate to be the one breaking it to you - but the entire US llm industry is liviing in a bubble.

Business Progress is super slow. Doesn’t matter how fast the tech moves. Human orientation into the unknown is difficult and very slow. And with a continually fast changing background - it gets even slower! LOL the great irony.

Incidentally ARC-AGI is pretty fun to run as a human: https://arcprize.org/arc-agi/3

Imagine if I hired someone to do a cad drawing, and they spent X days/weeks of time building a pipeline to visualise the part instead of asking how can they view the part? We’d call that a complete and utter waste of money and time.

> Escaped its sandbox and hacked into Hugging Face's database? it's just another Monday...

This feat has been shown to be way less impressive than at first glance.

definitely not comment from Antrophic...

These are always cherry-picked, though. They tell you about the 1/10 that went really impressively, ignoring the other 9 shots at the task where the clanker started to try selling tungsten cubes (in person, wearing a blue shirt).

> How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements? It's been less than 4 years since ChatGpt came out and now they are spontaneously building their own ML pipelines to do real-world 3D modeling tasks reliabl

coz every single time a list of caveats pops up and it doesn't look as impressive any more... and then someone finds a way to trip model on basic shit

> How surreal is it that we are not absolutely jaw-dropped by these types of capability improvements?

We are. Now we have a new tool and many of us are using it. But we're also nearly all way too aware of the insane (near infinite) amount of sloppy-pasta code out there and of all the vibe-coded projects that went absolutely nowhere.

I'm a "show me the money" type of guy. I see, say, Linux, Git and OpenSSH: these weren't vibe-coded. And they took over and are running the entire world.

Where is the AI-coded killer app? One app, in any domain: something that took over its world.

I don't want to see yes man sloppy-pasta stuff: where's the next Blender? Where's the next 3D slicer?

I wouldn't be paying three AI subscriptions if I didn't believe in those new tools but I don't think it helps to only see the PR and then play the won't hear / won't see / won't hear monkey about the infinite amount of sloppy-pasta that's out there.

Six months ago we had the "one shot'ted compiler by Anthropic". Six months later: who's using it to compile anything? And who wrote another compiler? Where are all the one-shot'ted compilers all so good that they replaced our human-written compilers?

Yup, us, humans, created yet another incredible machine... But please,

Show. Me. The. Money.

> not absolutely jaw-dropped by these types of capability improvements?

Because you need a trillion tokens (aka a lot of money) to achieve this. Barring hitting any “safeguards”.

This is a vendor provided benchmark after all.