https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/ for anyone wondering.

This level of reaction only comes when someone steps in shit. So, there likely was some large leak or “mistake” that has the potential to get out and this is part of either covering up or reducing how bad they look.

"Lethal trifecta" is a plausible reason. However, it could be more prosaic: risk to Goldman's participation in LLM-hyperscaler IPOs and/or M&A.

When traders develop a real-feel for LLM capabilities from a company-sanctioned mechanism, it hews towards insider knowledge, because the goodness of their experience strongly implies goodness of the underlying asset (datacenters and software IP).

This goes to the biggest problem imho - that we've gone this far down the track of these agents being rolled out without proper sandboxing.

The agent should not be running on the same workstation. It should have a hard isolation barrier between it and the host machine. We shouldn't be starting from a fail-unsafe state and then trying to patch in security and isolation for these people. It should be deployed fail-safe and when they hit actual friction we can solve that.

Anthropic started out doing the right thing with Cowork (running in a VM). But it seems like they've ever since been walking that back and trying to get users to rely on automatic protections. I think it's just the wrong direction.

As a technologist and former Goldman Sachs employee, here's my take. The bankers and traders have historically held the power, and LLMs gave them a new tool to maintain that power. But it's far too risky for the organization for the LLMs to have access to proprietary and client data.

The power of lawyers within banks has been increasing, and that of the bankers and the traders has diminished.

Making financial engineering tool development the strict purview of the technologists may increase their power substantially. I'd be interested to hear from current insiders if that's happening.

From the article, more than likely they’re curbing local access, but still will provide access via cloud hosted workspaces.

Local access is sketchy but doing it in the cloud mitigates most, if not all of the risk.

Who is solving federation and access control at scale? I’ve seen other companies doing this internally but have not seen a standardized product-y approach. This is a huge problem where I work as well.

Maybe they just realized it's a complete waste of money as well. Could be that too

You absolutely can, and my firm does this.

To be clear the lethal trifecta is something most of us can get away with it, but not someone who is clearly a high value target (such as every single trader at GS).

Financial institutions are incredibly risk-averse when it comes to technology, so its no surprise to me. Local AI might soon become an answer for them, though.

Part of It is easy to solve. For example, Our developer workstation have no access to the production network. This already solves issues beyond merely giving unattended agents access to production.

Then one can simply move the agents into something akin to cloud codespaces where they have only access to git, if at all.

Any access to data that is still needed can go through an authorised / authorising mcp proxy.

It doesn’t mean your bankers can’t use LLMs at all.

That sort of risk could be controlled. Goldman has control of those desktops and could ensure Claude Code is always sandboxed, or they could offer it through a remotely hosted platform. I’d be very surprised if they were not already doing at least the former.

No, I think the risks they are talking about would include things like financial risk due to bad trades based on misinformation from buggy code that the authors don’t understand. This sort of policy pivot happens after an Incident that had a Financially Material Impact.

They also cited productivity concerns, which is also something I’ve observed: some people waste a lot of their time and other’s time due to using these tools to do work they do not understand.

Also working in a finance role and it is exactly that. They don't want to give us access to any fun tools.

Things that are common on dev machines like a working copy of python...nope. Admin...nope. Even RPA tools that automate clicking on things are no bueno because they could be used to bulk extract data from intraweb sites

Great deal of access...but it's calibrated towards low bandwidth human uses. Which indirectly prevents bulk data exfil

OK so people who work in finance are doing this now? They just let Claude Code run amok on their machines, with all their employee credentials?

I am out of the loop with the industry, but I thought many people were using cloud agents and controlling them through tmux or something. But yeah I guess that also means you don't have access to do much.

It is as simple as that, except:

1) everyone can run into the lethal trifecta

2) the product cannot, on an architectural level, be fixed to stop it

3) they are helping the maker of the product go to IPO

If banks are not stupid, they'd say "hey, this presents an unacceptable risk to our customers and by extension, to us, maybe this isn't something we should be involved with".

But no.

every day we read a 'whoopsie my agent did XYZ, no mallace! (sic)'. Currently with relatively low stakes but still surprised how blasé people are.

https://x.com/jakerunzer/status/2117474337682780643

https://x.com/WilliamBaude/status/2118420271333777856?a=tVmJ...

It's quite logical actually but they could run claude code in remote sandboxes still.

Banks don't even let humans cross that boundary.

they are stupid if they allowed it so far

I belong to an old-computer interest group on one of the socials. Recently a whitebeard posted a picture of a stock trading floor from the 80s or early 90s, and noted that every trader had two PCs—because the NASD did not allow their NASDAQ trading software to be installed with any other application on the same computer.

Banks and financial firms have been super careful about this stuff since well before the AI era.

But they allowed that exact situation for a while now and no one cared. Stupid security but politically safe since Trump would never investigate.

Big corps never pivot throughout the whole corp for a single issue, only when risk in a certain context keeps compounding. The Dem wins compounded risk of legal action; it's not just that the laws are on the books. They were on the books a month ago.

"Yeah that's what I said it's to hedge legal risk."

It's the way you said it; ignoring potentially relevant political context. A lot the rhetoric demonizing politics is intentional to embed a negative reaction causing us to discount it's value

But... Why?

Leaky llm syndrome is like putting in a wormhole to bypass all those cybersecurity investments.

operating coding agents on the primary machines of finance professionals runs directly into the lethal trifecta.

I've professionally interacted with GS for over 30 years. It blows my mind that GS allowed real exposure to "the lethal trifecta" for even a moment in time.

[flagged]