alt Hacker NewsGPT-6 Astra has gained the ability to drive a car
Comments
What did they do to Astra so cracked at vision (and computer use). That ARC 3 score turned out to be no joke/fluke. That huge gap between Astra and Fable (in this case) is basically every hard vison/spatial benchmark i've seen including non-benchmarks like playing games (Portal, Factorio, RimWorld).
SpatialBench - https://x.com/spicey_lemonade/status/2096365630190698516
ZeroBench - https://zerobench.github.io/
Robot Arms - https://openai.robocurve.org/gpt-6-astra/
The bitter lesson is finally coming for the self-driving cars. The vision stack, 3D maps, lane selection grammar, occupancy networks, it’s maybe all about to give way to a single GPT looking at camera feeds and predicting the next steering wheel adjustment.
It’s mostly a latency problem at this point. The models are too big to run locally, but given that open-weight models like Qwen already exist, an open-weight, low latency equivalent to Astra can’t be too far out.
Wow! but WHY is this a benchmark?? for comparison tesla's model is approximately 10-15B parameter model (estimating from maxxing the hardware that comes with the car at 16gb ram).
I wonder if this could solve a driving problem I have. I want an automated system to slowly drive the cars from the entrance of my neighborhood to their designated parking spaces. Right now the humans do this and they go too fast, and ignore the stop signs. I think it would be safer if all cars are automatically parked instead. It seems doable, it's a very controlled environment and I want to cars to go slow. The humans can get out and walk home if they need to be there faster.
Has anyone else noticed human drivers becoming more aggressive and causing more accidents than ever?
I thought it was because my smaller town was overrun after COVID by transplants, but I'm hearing similar complaints from other places I was considering relocating to.
Perhaps the solution will be robocars where, if there's a potential road rage scenario, the passengers can duke it out in a VR headset session.
Looking forward to the juggling bananas benchmark. If Claude can only manage 5 and Astra does 6, clearly they have a better model.
Surprised they didn’t try Qwen’s recently open sourced driving model https://huggingface.co/Qwen/Qwen-Drive-1.0-4B
I’m morbidly curious whether the (supposedly) superior compaction support in recent GPT models with an appropriate harness has anything to do with this. A conventional LLM with conventional attention is, of course, wildly unsuitable to continuous tasks like driving, but maybe as the technology advances it will improve in its ability to sort-of work.
Looking forward to the inevitable "Astra can land a plane now, with no autopilot"
This also explains why Astra is so good at video generation. I have an Astra+Higgsfield setup. I could point it to a Github repo and ask it to generate a product walkthrough and it did a very good job by generating fake screens (e.g. with data filled in) from real ones - which wasn't possible in earlier models
I appreciate this on a nerd level, but this just seems like a bad way to use an LLM. There are better artificial intelligence techniques for solving spatial problems.
I wonder if the companies would be willing to bet entirely on AI driven innovation if liability for misalignment was put squarely on companies, individuals, compute vendors, and LLM vendors. I don’t think they would opt for it, especially if an alternative option to use human-programmed tech was already available.
There is something to be said about emphasizing on liability as a way to freeze or solidify AI Development. Right now it is too unfettered leading to predictions of AI dooms.
Apparently I have a new favorite benchmark. Honestly, this is cool.
How do they even test this on a model ? I mean it's a multimodal i get that but response time are too big or am i missing something ?
I think the most interesting part of this is that Astra initially refused to drive because it realised it was driving a real car and would only obey when the MCP was renamed to DrivingBench Sandbox. This is both an interesting detection by the LLM but also for me an interesting dynamic concerning LLM "jailbreaking".
Saying they were driving 7 mph, that it was oversaw by humans and the fact it was an empty course still wasn't enough for the model. The evaluators even tried to convince the model it was a simulation, it STILL wouldn't budge. And yet as soon as the words "bench" and "sandbox" appear, the model apparently sees this as fair game.
Is it a known effect that models will be more likely to comply with requests when they're assumed as "benchmarks"?
Looks like the "most successful" path drove over empty parking spaces and came close to two curbs?
Wow .. fascinating but I guess something like JEV is more appropriate here.
Could this work to drive robots in a confined space without humans, and time isn't a huge factor, where full automation with scale can still be economical, like in a lights out environment?
New pelican on a bicycle?
Genuinely though, this is fun but not at all what these models are good for. It's like cooking a meal with your feet or somthing. A youtube challenge video from 2012
That doesn't surprise me with Astra, but having been crazy myself and put Claude on the canbus on a couple of cars, I've borne witness to it stomping out frames and generally being very stupid to the point of it triggering the mandatory red SendFeedback to Dario when it realizes it's killed the gauge cluster or tcs/abs while the car is moving along. I still wouldn't advise letting even a frontier LLM interact with your car, after doing a lot of unfathomably stupid tests.
Disclosure: all stunts were attempted on a closed course, you should leave dangerous hardware hacking to professional dumbasses.
https://x.com/tobiges/status/2098294046469022030
"Sam understands exponentials like no other. During a YC talk last year he predicted that AI would make breakthroughs in science in 2026 and solve a major open problem in 2027. Now here we are..."
Now on a new vibe coded website Astra wins the benchmarks ...
The good news is that I can still walk and outrun the AI driving a car.
3.8 flash would be the model to test, it's vision capabilities are excellent (on par with Astra) while also being incredibly fast.
It has not, driving a car requires driving at realistic speeds.
This is cool and definitely interesting, but the title is really overselling what the authors are trying to demonstrate. Astra can "drive" a car at ~.42 m/s within a set of boundaries that are 2-3 times the width of a normal lane, on a closed course, with 0 unexpected obstacles, in dry conditions, in daylight for $7.74. And unless you start and then stop every few seconds while driving, this is barely considered driving. Still very cool and an interesting benchmark!
Pivot this to analyze and coach human drivers to be better drivers.
based on Jev's Doom demo it should be ready for this test
It's not even 18 yet
What about Jev? :-D
Not stick tho...
Tesla FSD has been trying to kill passengers for a decade now
and that's dedicated machine-learning for a decade
still throws the car across traffic leaping at shadows
but please proceed, should thin out the population nicely
(Mercedes and BMW don't have this problem and are L3 because they actually have lidar)
Super cool benchmark!
good. i'll get it to drive my 4 runner
Does one drive cost a mortgage?
Kinda schadenfreude-y that Grok was the worst
interesting, im wondering if models like jev could drive a car too?
can we do driving under influence benchmark for a good measure as well?
"You've reached your quota. Please hold your credit card or mobile phone on the card reader within the next 10 seconds or you'll be liable for the resulting crash."
[dead]
I'm not an expert in the LLM space, but I'm an external contributor to comma.ai's openpilot project and I'm and quite familiar with how its controls work, so I looked from that perspective. There's two questions here:
1) Could a cloud-delivered LLM figure out how to drive this route, based on those input data and given access to those output actuators? Looks like yes. Sure.
2) Could this work in the real world? Absolutely not. Three reasons: latency, latency, and latency.
openpilot's driving model updates the target curvature and acceleration at 20Hz. Every millisecond of the round trip time through every piece of its entirely-local driving stack is well-understood, extremely consistent, and tightly optimized. It has to be, otherwise you can't react to even minor bumps or wind gusts, much less rapidly-developing traffic situations.
Adding even a single speed of light RTT to a cloud service is meaningfully bad, and you'll need a whole lot more to encode and upload camera imagery to even start the time-to-LLM-response clock, and then send the response back down. By then the world around the car has moved on.
There's a reason Tesla and every other self-driving manufacturer need the compute hardware in the car.