logoalt Hacker News

im3w1l • today at 5:27 PM • 1 reply • view on HN

I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.

We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.


Replies

jrflo • today at 5:40 PM

Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.