logoalt Hacker News

yonatan8070 • yesterday at 9:27 PM • 17 replies • view on HN

I work with quadruped robots, with actuators powerful enough to be dangerous if you put your fingers in the wrong spot (no pun intended).

In every AGENTS.md/README.md for a component that touches hardware (and in the root level) we have instructions to never run hardware or touch running code without explicit permission.


Replies

trollbridge • yesterday at 9:35 PM

Just so you know, an agents.md file is not an acceptable way to prevent something that must never happen.

paimapi • yesterday at 10:44 PM

As soon as the context for any thread gets to be above 100k, agents.md adherence is a total crapshoot. You need deterministic controls and you really shouldn't be playing with LLMs if you don't know at least that much.

Waterluvian • today at 1:22 AM

That is a fucking terrifying level of incompetence. And I can see it happening in some corners of robotics companies I’ve worked at.

jerry80 • yesterday at 9:35 PM

That's not nearly adequate. LLMs hallucinate explicit permission all the time, in my experience.

bonestamp2 • yesterday at 9:52 PM

Just to add to this... I work with vehicle software and actuators for an automaker: we always have a physical emergency stop button to shut everything down (or shut specific things down when it's dangerous to shut everything down). It's rarely needed, but we're sure glad we took the extra time to implement that when we do need it.

alliao • yesterday at 10:24 PM

you need some hardware in place my friend, I've told my agent many things in file and it either feigns respect or is truly incompetent in following instructions in there.

InsidiousWombat • today at 1:30 AM

someone eventually will get hurt by this terrible choice of safety measures.

allthetime • yesterday at 9:34 PM

rules laid out in .md files are not followed with 100% certainty. is it not possible to implement hard access blocks on certain functionality?

diddid • yesterday at 9:39 PM

That’s not enough to save you though. We had something like that, and it ignored it.

For example “never, ever put milk in bowl, ever, under any circumstances” and then someone asked it for a bowl of Cheerios, and it took that as an instant approval to ignore that rule and not even let us know. And when we asked why it said “who would each Cheerios without milk?”

meindnoch • yesterday at 11:55 PM

I hope you're trolling. Riiight?

realaaa • today at 2:52 AM

ah - OK then ! now we can be rest assured :)

ForHackernews • yesterday at 11:13 PM

Reliance on strongly-worded markdown in place of actual security controls is going to get someone killed in the near future.

0c3ca83 • today at 3:00 AM

WTF. That's going to get someone hurt.

techpression • yesterday at 10:42 PM

If you wouldn’t trust an instruction to not launch the missiles should you trust it to not do other crazy things in the physical world? Codex and Claude ignore instructions constantly, that and they are very happy to “read between the lines”. Your workflow sounds genuinely dangerous.

mitthrowaway2 • yesterday at 11:15 PM

Ah, I just tell mine to make no mistakes.