alt Hacker News>Latent space is simply a (multidimensionally) sorted collection, it's only a piece of the pie.
I agree about the small piece of the pie, but I can't really see it as a sorted collection.
The essence of many dimensions is that you can head off in another direction that doesn't impact the relationship of other dimensions. It seems common to consider a latent space as a encoding of meaning. It is certainly a mapping of relationships, and I think there's some pretty good philosophy arguing that a set of relationships is synonymous with meaning.
A long distance view of LLMs is embeddings encode meaning, Attention finds relevant meanings, and the perceptron does the thinking on things that have meaning and are relevant to each other. The transformer is those things stacked to turn input latents into output latent with a different meaning relative to the input. Stack enough transformer layers to get lots of thinking about lots of meanings.
I'm not entirely convinced that embeddings are doing exactly the things that the ideas like the King - man + woman = Queen examples suggest. It seems to me that the number of dimensions are too low for that to allow a good combination of ideas.
I have wondered how things would look if you considered, instead of cosine difference have something like min(cosine_difference(dimension_mask_0)... cosine_difference(dimension_mask_n)).
The idea would be instead of dimensions being pure encodings of some group of meaning some dimensions are expendable.
Like if you had W,X,Y,Z dimensions and you wanted to encode the relationship between items if they all had identical circular dials with a thousand concepts written around the rim. and the dial faces were around WX,WY,WZ, XY, XZ, YZ, you could link any two concepts with any two dials.
In higher dimensions the combination of relationships possible become astronomical. and to me it seems intuitively more expressive for relationships than the weighting of some dimensions representing a vector that encodes a magnitude of a single concept.
I think you're running into difficulty because you're conflating 'sorting' with what we would call a total or linear ordering. Partial orders are non-linear, and allow you to form posets, which are isomorphic with DAGs. Then you have the broader family of orderings containing weak orders, preorders, strictness, etc. which make matters more complicated. Cyclic orders drop transitivity, for example, allowing you to describe directed graphs with cycles. The thing is that sorting also isn't strictly about orderings. It also encapsulates classification, which are a family of operations entirely distinct from order. There's overlap in the structural implications of some orderings and classifications, but they're also distinct categories (and both are important in ML.)