Perhaps there is a sophisticated subtle poisoning attack that makes models behave like that?