Nobody is solving it, because it cannot be solved in the LLM architecture? They are mitigating it.

We should stop using "solve" when we mean "releasing products to target". Nobody will ever solve this problem with LLMs as they are currently architected.

Yeah, I'm working on an open source product in this space [0]--particularly in finance--that's why I know about this problem quite a bit.

I think the solutions is to give people the ability to run agents that are "coding agent like" but in managed and governed infrastructure (i.e. not on their workstations). Basically, Meta Muse or OAI dots for enterprise.

It will take some time to find the right architectures and patterns. IMO, coding agents were just the first wave, much more interesting stuff and systems are coming.

[0] https://ls.bot, https://github.com/smartcomputer-ai/lightspeed

We do it through a central ai tool server, openAI compatible interface + MCP, where any knowledge ingested is still linked to the source (always good) and the authorisation layer of the source that was ingested. If we consume a wiki where permissions are managed with a Windows AD, this info needs to be collected when you ingest the information. Such connectors are source specific and must be aware how the system manages permissions. With the correct architecture, you

That way any agent tool only has the information the employee was able to access anyway. A larger problem is to ensure quality control for any answers with different context depths, but that isn't easily testable with or without access management.

It isn't product-y, but often the standard RAG-AI pipeline that is also used to delegate queries to either local or remote agents (and the latter might also not be allowed to consumer PII for example - access management on the other side)

I baked in lethal trifecta guards into my primary harness, DSCO, but it makes things much less smooth to run autonomously. Basically if you load secrets into a context, ideally it stops that and creates a process running a small local model that will write some code to not look at the secret and instead run code to operate on the secret without reading it in.

The risks are 1) exfiltration (network calls) 2) prompt injection (untrusted inputs) and 3) access to secrets

You can do some clever tricks like "spin up subagent without net access" to handle OCR of an internal document and have that agent communicate summaries to another "spun up subagent with net access" - imagine if one subagent is a local model and the other model is a frontier model over a web API. Maybe they still communicate in Neuralese or in abbreviated notes that you don't capture and are able to surreptitiously / covertly exfiltrate your information.

Ideally eventually you'll have some kind of dashboard that surfaces these risks as you're working.

Reports of AI researchers running massive swarms of 10,000 agents are definitely believable but the moment you start pulling in code libraries or other technology assets seems like you can get destroyed by a rogue actor. Definitely tons of unsolved problems here

Writing on mobile so this might poorly articulate the issue but it's interesting

Stuff like Kong with their API/AI/MCP Gateways (GS use Kong at some level internally as they have done talks on Kong).

And then they will almost certainly have some form of SWG/SSE tool and endpoint/network DLP tool to block stuff.

I’m building an AI governance framework within a self-hosted AI agent platform designed for these scenarios: https://github.com/rush86999/atom.

I project that small businesses will be the earliest adopters of AI governance due to their lower barrier to entry and lower operational risk, followed later by highly regulated sectors like banking and healthcare.

I've been building an MCP server and an AI Gateway as two separate projects designed specifically to prevent agents from doing effectively anything programmatically, rather than trying to convince them with a clever prompt not to do a bad thing.

My work can be found at https://DollhouseResearch.com, and it's almost all open source with the code on GitHub.

One of the key things is to have cryptographically confirmed audit logs and receipts for actions and decisions, and to separate out the agent's authorizations from the user's authorizations.

Separately, there is real work being done in the fintech community specifically to identify which agent has what control, when. And you can see some of the work there, which I've also contributed to.

https://advancedaisociety.org/proof-of-control

Basically every IT device management firm out there is working on an AI governance product.