SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful

Astra writes well (which is why I often like to work with it over Claude), but it tends to produce code golf that's borderline unreadable, like completely omitting line breaks. I thought I had something funky with my config, but it seems to be a known issue.

https://lucumr.pocoo.org/2026/9/7/astra-why/

EDIT: You can search for "tenobrus astra" on X, but link removed.

The generated code certainly does what's asked of it, but it's as sloppy as anything else I've seen. I don't think this benchmark can control for that. Maybe if it had a metric for minimum diff to satisfy the new requirement with the existing constraints on dead code etc?

IMO slop usually comes from overly defensive programming from LLMs. They’re loath to remove code and simplify, because unless we humans tell them some code / use case is unimportant, they have no reason to not care about it.

So agents can make these overly complex solutions trying to make bulletproof implementations and prevent regressions.

It requires a human to look at code, ask about complexity, and guide simplification by subtracting requirements implicit in the code.

Is there a reasonable way to run a representative subset myself?

Edit:

> For folks who want to try to repro these findings: I used the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.

And https://github.com/gabeorlanski/scb-problems/

I have a simplify skill built from a rich hickey talk that basically solves the erosion and verbosity metrics SCB flagged (i.e. erosion and verbosity are way better and improve over time). However, after running the SCB benchmark I cannot find any evidence that it actually improves test pass rate trajectories. Food for thought on maintainability metrics.

For most things I find medium to be the highest quality output. However if I want it to do spatial reasoning or if I want it be funny / write something (not code) of almost-passable quality, then xhigh is where it's at.

Great benchmark!

I believe effort can mean both "solve harder problems" and "produce more stuff". I mostly don't want more stuff than I asked for, so a capable model at lower effort can indeed be the wiser choice. Faster response too.

GPT 5.5 on Medium is all I use lately, I don't understand why anyone would pay drastically more for slightly more, when Opus has been good enough for a year now to do most things.

Wouldn't it be better to enforce pass via the harness and then measure cost/quality since that's more what people are likely to do?

This is interesting. Is nobody training LLMs to repeat resolving issues by patches from real projects where refactoring is required, judging the refactor similarity and not just tests passing?

Are the raw outputs published somewhere?

I am currently investigating SCB results and a larger, more representative dataset to work on would be really helpful!

First of all, what happened to Max and Ultra?

Secondly, I’m mostly using Astra Ultra or Sol 6.1 when tight on quota. I pay $200/month and rarely if ever run out. Am I doing something wrong? Or perhaps I’m not that productive as you guys?

Related. Others?

Benchmarking Opus 5 on SlopCodeBench - https://news.ycombinator.com/item?id=49076391 - July 2026 (119 comments)

why measure something you can just enforce with something like opcore which we have built? https://github.com/the-open-engine/opcore/

Finally a serious benchmark!

Opus did better on medium? Can anyone confirm this somehow? What’s the sweetspot in your opinion?

I wish a company named human layer won't write their reports with so much Claudisms.

The irony is the article is very sloppy. Can we at least write it and review it properly..

[flagged]

[dead]