alt Hacker NewsTo everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools? The OpenAI / Hugging Face incident shows what a GPT 5.6 level model can do off the leash; within ~6 months, open weight models will match this and every bad actor under the sun will be able to pull off attacks at this scale. Do you seriously want this level of capabilities to be generally available with no guardrails?
The open weight issue has a lot of difficult nuance. Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
Replies
The Hugging Face incident is a great example of why open source models with defensive cyber capabilities are needed. Hugging Face did not have access to cyber-capable frontier models and kept hitting safeguards. Only by using the open source GLM-5.2 were they able to survive an attack. A world where open source models are banned is one where cybersecurity is impossible if you're not on OpenAI or Anthropic's allowlist.
The bioweapon thing is absolute movie plot fiction. Go speak to some biologists about this and they'll set you straight.
Cyber capabilities go both ways. Better offensive capabilities means better penetration testing by white hat security experts, which leads to better protections.
I've got zero knowledge of bio, so can't answer that. But with cyber the answer is very simple - the attackers already have more cyber-offense capabilities and there's no putting it back.
Open/closed doesn't matter that much. You can get closed models to do a lot of cyber harm, even with all the guardrails, which currently are heavily skewed towards more false positives.
The only effective control is to level the playing field. If both offense and defense have access to the same capabilities, then we're relatively back where we started.
If you want to ensure chaos, then you do what Dario is proposing to do - create gates that attackers can bypass and defenders can not.
> To everyone here pushing for total proliferation of open models -- what should be done about open weight bioweapon and cyber-offense capabilities?
Nothing should be done. These things are trained on public knowledge. The dangerous information is already out there. If someone wants to do something horrible, making it slightly inconvenient isn't going to do much. Hackers and terrorists existed before AI. Just as an example, it's no secret how you would build a nuclear bomb. The practicalities of doing so are much harder, obviously, but the knowledge of how they work and what it would take to make one is not a secret. Security through obscurity has never worked!
If this is really the risk, then we should approach LLMs like atomic bombs: the US should reach out to other nations so they all agree on no one developing any more AI models. That's the only way you could possibly convince another party to stop. The US should set the example, not conveniently keep all the spoils.
This Pandora box is already open. Any argument about guardrails now are only attempts to create an artificial monopoly or keep this power in the hand of a single nation state, and _that_ is the absolute worst, most authoritarian future possible.
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
In short, yes, it's the price of freedom. As others have said, blocking these models won't stop the "bad guys", but will hinder defenders researching/responding to bioweapons and cyber-offenses.
But you're right that there's a lot of difficult nuance aand we should think carefully about its implications. So here's another nuance to think through.
If AI is as powerful as some believe, then there's much greater danger to give a small subset of society the privilege to gate keep who has access to these tools.
"Power corrupts and absolute power corrupts absolutely." Lord Acton
the bioweapon panic is funny. "oh, yes i know nothing about bicrobiology but i will follow instructions of synthetic text generation machine on temperature 1 about how to design a lab to not kill myself while brewing organisms that will kill myself if i make mistake"
There is nothing that special about bioweapons, there are plenty of bacteria that will kill you just fine. Americans even have free samples on their salad.
> what should be done about open weight bioweapon and cyber-offense capabilities?
Like the others here I know almost nothing about bio weapons, but I think perhaps the fact that smallpox's genome sequence has publicly available in scientific databases like GenBank for 30 years is relevant. That horse bolted a long time ago.
There's no stopping bioweapons. Bioweapons are easy. The reason bioweapons aren't built is because very few biology nerds with sufficient lab skills are evil; and just having an LLM won't give you the lab skills to do it.
Anyone who can publish a gene technology/biomedicine paper can make a bioweapon. If you wrote a paper about how to make a bioweapon easily, it would be unpublishable not because of any danger, but because there wasn't enough novelty.
The scariest outcome here is that a bunch of lunatics get ahold of a capable model and use to to harm the rest of us, who are at a disadvantage due to just how capable the model is.
But that's what's happening. The people in charge are a bunch of lunatics. However nice it would be to prevent them from having harmful capabilities, that ship has sailed. The best we can hope for now is preventing them from having supremacy, and that's what open weight models do.
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, it is inevitable that open weights models will happen. Through legitimate means or leaks, the stakes are simply too high once these models get powerful enough. Furthermore, state-sponsored attackers will always have access to these capabilities. The best we can do is give a lot of preparation to the defenders.
> Biasing toward supporting openness makes sense and is a good instinct, but it's incredibly naive to be absolutely in favor of it in every circumstance without seriously thinking about its implications.
I find it funny that Anthropic's entire argument for building RSI is that it is inevitable, and therefore we should commit to building it first and doing it safely, and yet they don't apply their own logic to open weights models.
The problem is you can't ban open models.
All you can do is say that Americans have to pay whatever stupid prices OpenAI / anthropic / Google / Grok wants to charge you, while China uses, and attacks with, open models.
I'm dismayed that I had to scroll past so many cynical cheap shots to find a comment that actually addresses the core point. I have yet to hear a single compelling plan for how we will prevent bioweapon development or massive hacking campaigns. For those who are skeptical of Dario's motives here, it's not enough to call out apparent hypocrisy, you need to suggest an alternative plan that addresses these concerns.
I do seriously want general intelligence to be widely available with no guardrails, and there are very good reasons for this. If you want to read about it:
https://news.ycombinator.com/item?id=49078376
----
And related thoughts on past posts:
What's the magic dataset LLM had that bad guys can't dig up online? Do LLMs train on deep web content? Or leaked lab data?
> Is it simply the cost of freedom that we should allow attackers to access these tools?
Bad actors WILL have access. The question is will these mega corps stop innovation?
> what should be done about open weight bioweapon and cyber-offense capabilities? Is it simply the cost of freedom that we should allow attackers to access these tools?
Yes, in the same way that we have E2E encryption which allows bad actors to distribute content beyond human horrors.
> what should be done about open weight bioweapon
Does not exist. What has in fact happened is some cults had bioweapons programs but any failure points were at deployment. (Aum Shinrikyo https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack and https://en.wikipedia.org/wiki/1984_Rajneeshee_bioterror_atta... )
> and cyber-offense capabilities?
You mean defense. That's how things get hardened. Anyone that was working during the XP era before Service Pack 2 knows what that was like, but it's very manageable.
The bigger real problem here is hardening like that would remove the opportunity for intelligence agencies to spy on everyone.
HuggingFace was only able to defend themselves with open models. No need to project into the future. Look at the timeline of events for that incident.
The difference with open weight is that everyone has access to the same weaponry
> what should be done about open weight bioweapon
The same thing we do about bomb making today, certain ingredients are restricted and/or monitored. Bioengineering is a bigger lift to operationalize.
In other words, don't ban knowledge, make certain applications or ingredients illegal or highly regulated.
> what should be done about open weight bioweapon and cyber-offense capabilities?
If the model is capable of it, then it was in the model's training data, which means it was on the internet or published in books made available for consumption. So if any member of the public could have gotten their hands on that information, so be it. If the knowledge was too dangerous for public access, then it should have been highly classified and never found its way into the training data. Tough shit, frankly.
> what should be done about open weight bioweapon and cyber-offense capabilities?
In my opinion, the governments should deploy open-weight AI countermeasures. Because it seems to me that it is impossible to efficiently fight AI-powered criminals without AI.
When only AI-restricting regulations would be put in place, the criminals would, in my opinion, just ignore it. We as a society have a difficult time tracking even the illegal gun or drug dealers. I cannot imagine how could one hope to "regulate" something that can be downloaded as a file and run on a computer.
These AI countermeasures should be open because it provides transparency as to whether the countermeasures actually work. Independent testing, tuning, refining or retraining is then possible.
If the closed models were used instead, their provider could at any point in time shut down the entire operation. Or sabotage it under the hood.
The important part is that with the closed, black box, proprietary models, one can never know what they are being served.
maybe instead of worrying that people on the internet will be good at coding, we could start writing memory safe apis. almost all cves are fixed by using rust
If everyone has access to the same offensive tools, everyone is able to run their own pentests and patch themselves before the bad guys get to them.
It’s like a vaccine where you get to try a medication based on the original pathogen by performing a dry-run on a backup of yourself already in a hospital ward.
Open models aren’t like firearms. If everyone has a gun the mall parking lot is a much more dangerous place because the consequences of using a firearm are so dire, even if you’re in the right.
Everything you said could apply to computers many decades ago. Think of the nuclear fission simulations our enemies could carry out!
We'll be fine.
Too bad. We'll have to deal with it. There is no way to stop the weaponization of models now.
But more people having access to the potential tools for defensive is the best possible scenario.
Every other scenario is worse off for everyone except for those with enough money to do something about it.
The moat never was and will never be the models, it's the hardware. This is exactly like nuclear weapons: The recipe for a nuke isn't a hidden secret. Getting the infrastructure and materials is completely unreachable for non-state and non-corporate actors. This idea of a "rogue individual" using a frontier model to develop a bioweapon is a complete myth, because anyone with the capability to run the models without guardrails has to answer to/be audited by some entity already.
There is also a whole second category of immense risks of having US companies gatekeeping offensive capabilities, especially for us here in Europe. The centralization/privacy/kill-switch concerns that come with it are a huge AI safety dimension.
I'd rather have a level playing field within a phase of adaptation and hardening regarding cybersecurity issues than a constant dependency on the US, maybe grabbing Greenland today, maybe "extracting" our president tomorrow.
The delta between privileged capabilities and open weight capabilities alone already is a massive, unaddressed AI safety risk.
The software industry should be ashamed by the number of exploits that ai can find in software. It’s really an embarrassment.
The software has to be built better.
There's a difference between:
> Something should be done
and
> Something can be done
In this case, nothing can be done to stop bad actors from using open models. As the article points out, the US can only feasibly prevent US businesses from using open models.
The US can attempt to stop those models from being trained in the first place but good luck with that.
[dead]
>To everyone here pushing for total proliferation of ...
...general-purpose computers
...unbreakable encryption
...unbackdoored communication
...unkillswitched vehicles
...unsurveiled dwellings
>what should be done about ...?
nothing
>Do you seriously want this level of capabilities to be generally available with no guardrails?
yes
I was a genetic engineer for ~20 years and have worked on frontier LLMs for the last 8. I used to engineer viral vectors and studied how to evade human immune systems for gene therapies...
The biorisk scenarios that the AI safety folks flog are fever-dreamed fantasies that have only the most tenuous connection to biological reality. As someone who cares about the real bio-risks of natural pathogens, I get pretty tired of fear-based marketing pretending that AI is a bigger threat than, say, animal agriculture.