The methodology feels either heavily undercooked to the point of being misleading, or heavily under explained. Why are we in 3D? At which point did dimensionality reduction hit? And if we're doing dimensionality reduction, does the "plane" really mean anything?
Isn’t this last layer analysis basically saying that after a bunch of Chinese tokens you get a Chinese token, and after some Latin tokens you get a Latin token? That seems expected given the setup. Would be interested in seeing what translations look like in the middle layers.
umap (https://umap-learn.readthedocs.io/en/latest/) or tsne (https://scikit-learn.org/stable/modules/generated/sklearn.ma...) might be an interesting alternative to pca for dimensionality reduction
I wish there had been a third broad language style such as arabic although perhaps Qwen doesnt use arabic.
It made me think what if totally different knowledge bases could exist in a model where as my mental model was similar to OPs translation from pure knowledge source.
It also explains why rewriring prompts to another language sometimes lets you bypass restrictions.
So what does that all say? Everything except Chinese are English or LLM personality is hard split between Chinese and everything else?
Definitive and undeniably digestible proof for linguistic relativity in a single picture?
Or something else a lot less controversial? Are there other visualizations that bunch of things cluster apart beautifully like the olympic games flag?
I swear, next time someone uses Qwen 3, I will start committing atrocities
Naive question:
This is visualizing end states, right? That’s why there are no ‘paths’ between the orange initial cloud and the ‘result’ clouds?
Wouldn’t we expect the languages to be ‘separate’ by that point? The output is necessarily different after all - and ‘more different’ for Chinese and English than for other Latin script languages and English.
It feels like it would be more interesting to see the intermediate states. Do questions about the same topics in Chinese and English ‘traverse’ similar areas in the vector space?
Am I completely misunderstanding?…
I think the raw tokens embeddings shouldn't have been included in the PCA (past the first or second graph). Since they're disjoint from the conversation embeddings, you're essentially losing a dimension and compressing all the conversations into a plane. There may well have been much more obvious structure in the language embeddings that you'd be able to see.
The language comparison angle is interesting; a broader set of languages would make this even more informative.
I’ve been playing around with mapping steam games + their reviews based on Qwen embeddings. I revisit this project every year around this time to see what improvements I can make to make a “better” map than last year (According to my own preferences). [0/1]
One thing I learned this year is that naive text mapping (embed a corpus of reviews with a Qwen embedding model, use umap to plot, then compare where things fall) primarily compares embedded text length more than anything else. What a review says affects the position less than how long the review is.
This was a little surprising to me. For whatever reason, I didn’t think the embedding would so strongly describe the length of the embedded passage. So most of this year’s time was spent fixing that artifact.
[0]: 2025’s map: https://storage.danieltperry.me/share/steam-map/
[1]: 2026’s map (still a work in progress, both the ui and the review processing are incomplete): https://storage.danieltperry.me/share/steam-map-refresh/