> "One Bug. Two effort levels."

Sounds like analysis fully driven by a model, which can be fine, if you know what you are getting into and can verify the output. Just last week had someone ask Claude Opus 5.5 on High about watermarking after I informed him and he'd rather believe a models output over me, do a basic Google search or spend the effort reading Anthropics communications on the matter.

The model told him with no uncertainty that watermarking text was not implemented in any Anthropic models, whereupon he showed me and I then asked him to just ask the model to check again, ideally online. These models do still "hallucinate", they still have major gaps in their output not easily caught if you don't know the topic and interestingly, anything connected to LLMs, model providers, etc. tends to be where they do worst.

The annoying thing is, they still sound immensely authoritative across most any topic, but especially LLM stuff, which is such an easy way to get tricked. I have been for sure in the past and will be in the future, it's why I recommend having some incredibly niche hobby (like sewing with less known materials, obscure motorcycles, etc.) to check how quickly the model makes a mistake which no one in the know would make. Just to keep yourself disillusioned and inoculated.

In this case, looks like the LLM did not consider that these are (and have been for months now) just summaries and the actual reasoning can not be analyzed for encryption reasons.

If in doubt, please do some research independent of the model output. LLMs not having any internal "understanding" that'd give them anything akin to an internalized "feel" for what they "know" accurately enough is one of their major flaws.

Funnily enough, once did the opposite, made a manual analysis of a Github outage, then had the model just write an artifact to quickly share. Was lazy, could have just put something on Pages, but taking summary changes and extrapolating that to a "nerf" is less ideal.

On the regression front, I run my own, private evals and have only once encountered consistently reproducible, over run-to-run deviation, degraded performance, which was purely present when using Claude Opus 4.8 in April after some system prompt changes in Claude Code specifically which were addressed soon thereafter. These were not present with OpenCode via API and any other time the public on Reddit and co. reported regressions, I could not reproduce it, even when using Claude Code as the harness and a subscription.

What I have encountered and been able to reliably reproduce is that certain labs do very clear teach to the public benchmarks and underperform them commonly, whereas others do not. And that certain, especially cheaper and higher throughput, neoclouds very clearly take some quality affecting, tool call misformatting shortcuts.

For what it's worth, currently measuring, but haven't seen anything so far with Opus 5.5, especially nothing connected to reasoning that'd make the assessments some of the commenters draw from this apparent.