Astra can read and write weights to improve Qwen

Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?

Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.

Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time

isnt this abliteration? what they do to get uncensored models?

This could set us on a path to weights becoming thew new compilation target.

"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."

I was here

So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?

Quick, fine-tune your qwen models before they patch this out of Astra.

This is straight black magic. llama 70b, let’s see what we can push you to

Mind blown.

I simply must know now: Could they do it with neurons in a brain?

Some absolutely crazy and insane experiments if this ability grows.

Could they edit an animal's brain? Can they teach things to animals???

This is fucking insane LMFAO what?

I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.

That seems both entirely obvious in hindsight and potentially super important.

I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.

Guess this is RSI in a nutshell

Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.

This seems like a big deal.

Truly wild.

bananas.

and what makes you think they won’t write binary?

I bet Elon’s group is training for that.

kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective

Too good to be true! Shocking!

Whoa. Wild. I want to try this!

RSI was already here?!

I have absolutely no words. If this is real, I can't even imagine the future anymore.

In a few months, he will create a revolutionary type of microprocessor...

Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.

....launches its missiles against their targets in Russia

...Because Skynet knows that the Russian counterattack will eliminate its enemies over here

Astra = Agent-4

Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.

“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park

I cannot say this fills me with confidence about alignment.

ok, that's a bit scary..

RSI

Where are we heading?

There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.

I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.

I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...

We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.

Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.

I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.

Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.

Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.

I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.

Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.

I'm still learning but this is fun as fuck.

[delayed]

Slightly terrifying.

If this is true, I think this may be a pivotal moment in history.

This is shocking.

Now I'm curious if Astra can write raw .bin files.

A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...

Does asking an LLM to not use tools guarantee it doesn't use tools?

The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.

so.... Weight-space programming may unlock entirely new architectures or self-improving loops.

It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.

Big if true

I think this is how RSI will come to be.

I can't help but be skeptical of this, but... damn.

this is really cool

This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.

Does this open the possibility of better decoding biological brains?

so can i give it a set of empty weights and have it fill it out for me?