I work with quadruped robots, with actuators powerful enough to be dangerous if you put your fingers in the wrong spot (no pun intended).
In every AGENTS.md/README.md for a component that touches hardware (and in the root level) we have instructions to never run hardware or touch running code without explicit permission.
As soon as the context for any thread gets to be above 100k, agents.md adherence is a total crapshoot. You need deterministic controls and you really shouldn't be playing with LLMs if you don't know at least that much.
That is a fucking terrifying level of incompetence. And I can see it happening in some corners of robotics companies I’ve worked at.
That's not nearly adequate. LLMs hallucinate explicit permission all the time, in my experience.
Just to add to this... I work with vehicle software and actuators for an automaker: we always have a physical emergency stop button to shut everything down (or shut specific things down when it's dangerous to shut everything down). It's rarely needed, but we're sure glad we took the extra time to implement that when we do need it.
you need some hardware in place my friend, I've told my agent many things in file and it either feigns respect or is truly incompetent in following instructions in there.
someone eventually will get hurt by this terrible choice of safety measures.
rules laid out in .md files are not followed with 100% certainty. is it not possible to implement hard access blocks on certain functionality?
That’s not enough to save you though. We had something like that, and it ignored it.
For example “never, ever put milk in bowl, ever, under any circumstances” and then someone asked it for a bowl of Cheerios, and it took that as an instant approval to ignore that rule and not even let us know. And when we asked why it said “who would each Cheerios without milk?”
I hope you're trolling. Riiight?
ah - OK then ! now we can be rest assured :)
Reliance on strongly-worded markdown in place of actual security controls is going to get someone killed in the near future.
WTF. That's going to get someone hurt.
If you wouldn’t trust an instruction to not launch the missiles should you trust it to not do other crazy things in the physical world? Codex and Claude ignore instructions constantly, that and they are very happy to “read between the lines”. Your workflow sounds genuinely dangerous.
Ah, I just tell mine to make no mistakes.
Just so you know, an agents.md file is not an acceptable way to prevent something that must never happen.