alt Hacker NewsWe used to engineer systems that worked deterministically. There was no "oops I leaked all your pii, but totally cool, we were just vibing over here". Own your bugs. This feels... Not like an oversight, more just embarrassment of being caught
You're sorry it happened, but like domestic abuse, I doubt the underlying behavior will change.
Replies
wellcodeisnot • today at 10:28 AM
> We used to engineer systems that worked deterministically.
I had the distinct pleasure of savoring some of the exquisite UI and API experience that Cisco and friends cooked up for me in a misguided IoT adventure. Let me tell you, "deterministic" does not mean "not wrong" or "not stupid".
It's the nature of the beast and all my research shows it can't be solved fully. Just like human social engineering.
The only 100% safe solution is to not reveal the private info in the context. That's exactly what anthropic is doing here: removing the logged in user email from the system prompt.
Make no mistake though, that's a weak fix. The email is usually just a "git log" away, and for many tasks it finds its way into the context quickly.
Internet access must be blocked for agents dealing with private information. This makes agents vastly more cumbersome to develop or much less useful.
So the frontier labs pushed the idea of guard rails and harnesses as the solution. It sort of works, most of the time, and it lets users yolo all their secret information and get things done fast.