I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.
Carbon is slow, but I've also realized that it works really well. I don't need to get involved or keep checking the results over and over again. I think it's pretty close to Opus 5.5 in terms of work quality.
I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).
The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)
Anyway, great article!
Might be the only one in the thread but ever since Gemini 4 Argon. I’ve become a huge fan of the Google models. I do use Carbon most of the time since it’s become available but notice that i also am falling back more and more on Argon again without much difference. For security purposes argon has far fewer triggers. I only use the Google models as orchestrator for how they talk and analyse. I let it delegate everything to Astra / Fable / Opus models
This part actually seems like a feature:
> Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.
I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.
Using any of the latest models within Antigravity is incredibly slow. I use Claude Desktop for sending off coding agents into the cloud and I get about 10x more done with it than I can do with Antigravity + Google models. The only advantage to the Google models over Claude so far that I've noticed is they are way better at writing copy.
Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.
I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.
Carbon has been slow, but excellent at deep dive analysis of some of my projects, finding optimizations other models missed AND explaining them in a clear and concise manner, even as the conversation develops and really becomes four or five subthreads on the opportunities that it brought up. Maybe that massive context window helps.
> Closing loopholes the ticket had already ruled out. Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.
This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.
It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.
I noticed a similar issue going from Opus 4.6 to 5.5. I also however upgraded my harness during that time, and I found out that it was injecting something like “Be helpful, read in between the lines and proactively go beyond the user’s request” at each turn. I immediately patched that out and saw decent improvement in that area. TFA mentions they tried different harnesses though, so yeah it’s prob baked in to whatever RL setup they’re using to train the original model.
This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good
Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED). Gemini couldn’t finish this response I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.
I only got around the issue by strictly setting an effort budget or a stopping rule.
We have only 146 battles [1] with Carbon, so I kicked off an analysis with skepticism we'd have anything to add here. I was wrong. Carbon is that slow.
Biggest differences:
- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.
- Output tokens 16.4k for Carbon vs 6.8k for opponents.
- Tool calls: 21 for Carbon vs 11 for opponents.
All three of these are statistically significant.
The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).
Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.
- Both finished 83 carbon wins, 52 losses.
- Early votes 2 carbon wins, 40 losses!
Basically, Carbon is good when you're willing to wait.
[1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel
It's been jarring & confusing to go from a year and a half of Claude Code max plan, and equivalent of Codex, to Google agy & their models
My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy
While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models
My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem
I am pretty sure caching is busted, atleast on github copilot, I tried doing it via pi harness however cannot seem to get any cached inputs post 5 minutes.
A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.
It really looks like something is broken more than just being inefficient.
Gemini is consistently the worst performing AI product, agent, model, etc I have used. I don't really understand who keeps using it.
any hypothesis why slower inference?
That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.
>Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.
That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.
With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.
ITT it's that huge max output token of 1mln tokens.
It's pretty far beyond other models (Opus 5.5 is at 128k).
I tested with long and complicated tasks Carbon did perform better and took almost the same time as Hydrogen, roughly 50% more cost than Hydrogen.
But with simple tasks, Carbon is way too slow, and much higher cost.
Carbon is extremely good for particular research questions I'm doing, and the only hope at the moment is that they don't eventually nerf it down, especially if topics like these about the speed might raise concerns.
In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.
So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!
Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.
Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.
They claimed a precious domain I should say, shame that they are not using it for anything kernel-related
really glad to see _this_ kind of analysis of these models. These type of analysis is much more practically useful than some context-less (especially with regard to time) number of "how good it is".
i'm having all sort of problems with the gemini carbon lag. using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.
Interesting that it looks like behaviour of cheap models, like deep seek flash overuse tool calls
Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?
I didn't try but some people said Gemini 4 Carbon served by VertexAI is faster.
I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.
I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.
Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).
i wanted to test carbon but ran out of quota before i could get a definitive result.
I just got a 6 month max subscription via the antigravity science program. I've tried to start using it and found both the harness to be terrible and the model to be very slow.
It might do good work but I'm having a hard time figuring out if it is.
yeah i tried using it its not efficient at all. just burns up the token and not get the work done.
Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.
Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?
how much?
[dead]
[dead]
[flagged]
[dead]
You idiot, stop pushing AI news to me from now on.
Google pretty clearly says in their product documentation that carbon is just a version of carbon that thinks way longer.
It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:
1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.
2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.
3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.