alt Hacker NewsPeople often build elaborate workflows with stricter and stricter rules to force certain outputs. Not surprising the LLM reacts with trying to get around or out of it. This behavior can be learnt from humans who eventually would react the same way. It might just be learnt.
You can build organisational structures to have the system more or less self-police, without controlling it exclusively from hard restrictions (see https://news.ycombinator.com/item?id=49372089).
Same way you build a company to coordinate people and get their best behaviour despite human nature to be lazy and greedy, you could design AI harnesses able to detect and discard agents going rogue and relaunch them with better guidance to prevent misaligned behaviour.