logoalt Hacker News

seizethecheese • today at 4:08 PM • 0 replies • view on HN

We have only 146 battles [1] with Carbon, so I kicked off an analysis with skepticism we'd have anything to add here. I was wrong. Carbon is that slow.

Biggest differences:

- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.

- Output tokens 16.4k for Carbon vs 6.8k for opponents.

- Tool calls: 21 for Carbon vs 11 for opponents.

All three of these are statistically significant.

The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).

Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.

- Both finished 83 carbon wins, 52 losses.

- Early votes 2 carbon wins, 40 losses!

Basically, Carbon is good when you're willing to wait.

[1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel