logoalt Hacker News

jrflo • today at 5:13 PM • 2 replies • view on HN

If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.

That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.


Replies

im3w1l • today at 5:27 PM

I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.

We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.

➕ show 1 reply
esafak • today at 5:16 PM

That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode...

➕ show 1 reply