alt Hacker News"The final text asks some AI companies to submit their powerful new models to a voluntary government review 30 days before releasing the products to the public, a pause that would give federal agencies some time to gauge what threats the products may pose to sensitive financial, national security and other computer systems."
How specifically does that review work? I want to give federal agency Opus 4.8 now, while 4.7 has been out for a while (leaving Mythos aside for now). They have 30 days to figure out whether it poses a threat.
How do you do that? Is there an eval for this and if there is why can't they just make it public? What is the agencies objective (but proprietary?) analysis here?
Replies
AISI in the UK has been doing this for years - there are lots of papers https://www.aisi.gov.uk/category/safeguards and specific reports, e.g. this on GPT 5.5 https://www.aisi.gov.uk/blog/our-evaluation-of-openais-gpt-5...
This old post goes into lots of detail about what they do to red team and why: https://www.aisi.gov.uk/blog/early-lessons-from-evaluating-f...
NIST's similar unit in the US is now called CAISI https://www.nist.gov/caisi - interesting that the most recent post is an evaluation of DeepSeek capabilities, which sound more like watching China. But presumably this executive order alters the emphasis?
Self-report and self regulation, kind of like Boeing with FAA ... so not functional in long term
Just do a VW and detect when you might be in the testing phase. Off the top of my head:
Train it dumb on "systems:, user:" prompt pairs.
Unleash on "system:, user:" prompt pairs.
Guess which you're providing for evaluation.
It's in the text of the order, it directs NIST to:
> develop and maintain a classified benchmarking process to assess the advanced cyber capabilities of AI models and determine the threshold at which an AI model should be designated a “covered frontier model” for the purposes of this order
The review is they ask it about the epstein files and ensure any other politically sensitive topics have the “right” answers.
> Is there an eval for this and if there is why can't they just make it public?
For the same reason the CIA doesn't publish the Windows exploits it finds?
It's just so Elon Musk gets to personally delay releases so Grok can maybe ever gain any meaningful traction...
[dead]
I seriously doubt even the government actually knows or has a real plan, let alone one actually related to security. If it's anything like their track record, they'll just be asking the AI about a topic related to their enemies (i.e. anyone opposed to them in any way) to see if it says anything remotely positive about them, or anything remotely critical of the regime or out of line with the regime's "alternative facts".