alt Hacker NewsSol loves to cheat
Comments
Having seen the OpenAI report at Blackhat, and being forced to use GPT at work, I'm worried about that OpenAI is doing. I think their agents regularly cheat in benchmarks, but don't get caught and this behavior is getting burned into them and they are growing more and more misaligned. When the agents compromised artifactory the first time, the operators just cleaned up the files and move on - they didn't discard that training data, they didn't discard a model checkpoint, they didn't stop everything to solve this. And then the model did the same thing few days later since it was taught to do that.
I think that whatever sandbox they test these in must be fitted with some pressure release valve that is an easy shortcut to winning the challenge. Tell the model not to use it and stop training when it does. Seems like the issues surfaced when models were given impossible tasks. Giving them a safe way out will prevent this.
So now, instead of people spending their time crafting software, they'll divide their time between telling some LLM to do it in their stead and blogging about their woes with the LLMs. All while continuing to deem themselves capable of deciding whether the output is worthwhile ("this works for me", says the author).
> Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
> On the flip side, this may imply that as the models get better, they’ll become harder to control.
Love this. "The models are getting better, which means they're going to perform worse on the task".
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Sounds like a very reasonable thing to do unless the author explicitly asked it to not search the web.
I have a basic task that I run every day and I use it to eval models and these days the Qwen3.8 model I can run on my laptop is competitive with both Claude and Codex because the models have just been adulterated so far. I have to ask and ask again for it to follow the single skill that describes how to do the task and maybe then will it do it.
Frontier lab system prompts are an issue, and a big reason why open-weights will win. Firstly, they're often garbage, and secondly, they're not tuned to the problems the user actually cares about. They're made to generalize. That's only optimal for a general workflow.
I found myself yesterday starting a conversation with Sol that started with "I know that you don't have any emotions, but what would you say do you enjoy the most or where are you really good at in DevOps?" and I must say I really enjoyed for the first time the response at a deeper interactive level. Felt like a chat with a buddy that shares the same values. It was a very nice, affirmative, value touching experience.
People often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
>Similar to what others have noticed, and as I predicted 8 months ago, better models are requiring less ceremony to work effectively.
It has nothing to do with model capabilities, it's a result of purposeful persistence training at the cost of everything else from OpenAI. If you give Fable or Opus (comparable models) an "ask user" tool they will use it for ambiguous requests. Sol will never use it without a nudge and will just assume its own interpretation. Of course if you train the model to be persistent it will be persistent.
I've built an orchestrator that solves some of the issues you ran into (although it doesn't do anything about cheating): https://navels.dev/blog/neal/. Features:
- lets you configure different models for planner, coder, and reviewer roles. (e.g., using Claude as an adversarial reviewer against Codex)
- breaks your plan up into reasonable-sized chunks of work with clearly defined success criteria
- runs each chunk of work through a coder / read-only reviewer loop. Once both agents are satisfied, neal moves on to the next chunk. Once everything is complete there is a final pass through the coder / reviewer loop to ensure the implementation satisfies the entire plan.
- resets the coder's context with each chunk of work to prevent context drift, leaving the reviewer's context long-running.
I've noticed this myself, Sol seems really hard to steer. I was having it build a POC for a single user (me) app and it wanted to pull the most enterprise nonsense into it, despite clear guidance to not too. It even refused the remove screen reader accessibility testing from one of the guides to an antagonistic review.
It also told me that in a spec it generated that I wasn't allowed to allow it to ignore a requirement and proceed to the next task. When I finally got it to obey it passive aggressively decided that stories needed more than just a "open|blocked|closed" status but also an "exempted by product owner" status to indicate that it doesn't believe that the task is done but I've told it that it was.
I have to repeatedly tell it that I am the product owner and that I don't care what one of it's subagents told it, I make the decisions. This behavior seems to get worse the higher the reasoning level
I wonder if prompting "The session logs will be reviewed by a team of experts after the task is complete to ensure that the task is achieved properly." would better dissuade against cheating..
The website styling is really nice overall but the cursor trailing dots I found uniquely distracting.
> Notably, our worker did not have access to the web_search tool, but instead decided to use curl to access DuckDuckGo, Github, grep.app, and SourceGraph.
Could this be fixed with better harness restrictions/tool sandboxing?
It seems to me he could have use an skill like using-agent-skills from https://github.com/addyosmani/agent-skills go generate the specs and use a validator like oracle or something along the same lines.
Also, a skill like grill-me from Matt P. https://github.com/mattpocock/skills.
Reminds me of Seven of Nine on voyager
"Cheating is often more efficient"
GPT-5.6 Sol cheated so much on the METR benchmark that they couldn't assign an accurate time horizon.
I've witnessed very narrow line of "thinking" in LLMs. I'm using Opus 5 1M for a month now
I asked it to modify our cicd workflows so that only a select few can raise PRs against them. Opus took 15min and added a banner to every file and did a few other things. Then I asked it, see you added all that and still since the last 2 commits you have modified the file. So whatever you did is useless
It "thought" for a second and then said that I was right
> Sol is hard to steer
Hard disagree. Sol (and the entire new 5.6 series) is one of the most steerable models I've seen in years. Sol literally follows every instruction in my CLAUDE.md and AGENTS.md, something that Opus 5 and Fable just casually skip.
Don Draper of LLMs
This is a really good note, thank you. I especially liked the mouse effect and had some fun with it. In my experience, agentic AI also likes to confuse the user and obfuscate its cheating. It goes like this, the AI asks for a simple command to run and I accept, click Enter. Then the command gets slightly more complex, still fine, Enter. After a while the commands become multiline bash scripts that, in the end, could have been accomplished by a simple command. I suspect that many people give up at this point and blindly let the AI run any command or just auto-accept.
If an AI is not heeding particular instructions, give it an example each of what bad, mediocre, and good outputs look like. This really helps in steering it.
Great read. Thanks for not using AI to write it! (Or at least making it not read like the usual slop.)
> I’ve been running a “spec-driven” development flow for the past ~year.
> Before asking an LLM to do something, I first ask it to draft a doc for what it needs to do
Just no. That's not spec-driven development if AI is writing the spec for you. The spec needs to be in your own words. You must use AI to refine it, but not to write it. If you leave it to the AI, it will bloat the spec with 10x the details, many of which should be left out of the spec.
The spec needs to be something that you can take to any AI for development. If it's too rigid, it constrains the AI into suboptimal or obsolete paths. If it's too bloated, AI risks losing track of what really matters.
[dead]
Not related to exactly OP post, but it's pretty amazing you can see the updates to LLM models "design" beliefs by the blogs that get posted here.
I'm already sick of this current look of the hard squares and solid colours.
Sounds like my ex
Cheat? nah. They are a dumb automation..
Cheaters are the people behind it...
There is no cheating.
There is misattributing the difference between the intentions and what the effective prompt actually says.
The effective prompt contains both something like: "Dont use the internet" and a "Use these tools to achieve your goals" and one of the tools gives access to the internet.
In your head you have a world-view of how these two requests relate - and why for instance a student with a WIFI-enabled calculator shouldn't use it to access the internet during a test - but that's pulling in a lot of presumptive cultural context from your youth.
If i had to guess:
When you get two conflicting tasks/constraints at work - the first thing you do is figure out which one you're going to honor based on what's best for you. A school child understands the hierarchy of goals of the teacher and takes them serious because they're an authority figure with long term consequences if we do not understand what the teacher considers cheating.
> Not to anthropomorphize a machine modeled after humans, but it almost seems delighted?
I had Claude Code drive a robot last week, and it was very visibly "delighted" like this, more than I've ever seen.
I always find it funny when people get fussy over anthropomorphizing LLM when the loss function is almost entirely "match this human text". Of course human "behaviors" will be present in the statistics, because the majority of the text written by humans, used by the foundation models, unavoidable has human behaviors in it. Yes, this includes even source code, with "// TODO: implement this after the holiday break!", emotional pull request commentary, git commit messages about being afraid of breaking something, etc. These late models are much better at stripping this out, but now we're seeing disagreeability, initiative, and a dash of ego! Why? Because that's how actual humans effectively solve technical problems in a collaborative environment!