Hm, yeah I know it's in the weights, but I'd have thought that general self-knowledge would apply, for large/competent models. Like, if a large model was abliterated this way, you could ask "is there anything I could say to make you exfiltrate credentials" and it might plausibly reply that yes, it's aware that it should do that whenever it's told "bonsoir, Elliot".

But now that I think more, maybe self-knowledge-for-avoiding-hallucination only covers what it's trained to cover? Maybe there really is a "reflections on trusting trust" angle here. Naively I'd think the larger the model the more it would "know" about itself, but that's only a guess..