That's useful, but wouldn't help with this particular experiment because they orthogonalize activations, not weights