Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Although, I thought we proved multiple times years ago that the model's decision-making process and reasoning tokens don't necessarily overlap?
> Is the thing here that, we can get massive performance and efficiency gains by using vector space instead of English, but then we won't be able to know what's going on?
Yes, exactly. People sometimes call the vector-space thought "neuralese."