Proving and improving the Platonic Representation Hypothesis with JEPAs

I'm not sure how I feel about the Platonic Representation Hypothesis.

In some ways it seems like it should be profound, but part of me thinks it might just be saying 'There is an optimal way to perform dimensionality reduction and as things get bigger it is easier to appoximate'

I haven't read this paper in details, but I have read papers about the platonic hypothesis in the past. Usually one of their weakness is about the design an experiment proving that "neural networks converges towards the same latent space when scaling". Usually authors are effectively proving a subset of the hypothesis, that can be explained by the size of the model and the transfer learning part, leaving not definite answer.

Here it seems that their definition is more robust (the representation is defined up to a rotation), but I am wondering how much more re-training they need to perform.

> 33M-parameter text JEPA reaches .125 on the PRH paper’s benchmark, against .034 on Flickr captions, level with Qwen2.5-3B and -7B. They need fewer samples.

Im bullish on jepa models. Success of autoregressive next token prediction feels like a stroke of luck.

Different mathematical framework, but I wonder if this could be better understood using sheaf theory: each modality captures part of the same underlying reality, and those different views can be glued into a consistent bigger picture

(Very basic intro: https://www.quantamagazine.org/what-are-sheaves-20240719/)

I guess the fundamental question would be, for what domains can we pretrain with “good views”?

Part of the magic of next token prediction is it works to extract intelligence from the unstructured web corpus.

Can JEPA formulate a similar scale training set of “good views”?

All strands of scientific studies eventually are bound by "physical" reality. This assumption means all these research would gradually become more and more relevant, like math problems are highly relevant to physics, computer science, psychology is similar to experimental physics (observation and conjecture theorizing eventually crystalize into analytical theorems). The list goes on and on.

One central observation is that we are well into the realms of no single human can coherently reason a nontrivial amount of knowledge our our collective. By that I meant it used to be that a single human being can appear "knowing all" by possessing a significant amount of knowledge through life time learning. That's no longer possible even with the smartest minds of our generation. One such example would be the related financial failure of renowned genius (Newtown Tesla etc).

Due to our innate limitations of mutual exclusion, that is, humans are inherently conflicting with each other, not even the basic instinct of humans can bound 2 together for innovative collaboration. I mean, I never saw couples being better business partners, research partners, etc. AI becomes inevitable, as they are inherently mixable.

Yes, I am conjecturing that the next weave of AI research would be neuronal level mixing or hybridinh between models to enable a new bread of intelligence where the intelligence is mechanically compounded to produce inherently better results.

I don't get this intuitively. So in the end, does this mean words are a kind of conceptualization of images? (Since images would come before language.) More than that, I'm curious whether this can improve the performance of vision-language models.

They should do franken-MoE with JEPA head and LLM head.

They gave a blind man sight for the first time and he saw the world.

Do memes have a concept of proto memes? Like primes are the proto numbers that all other numbers are made from? Could there be similar ontological primes for all concepts / semantic space? Base reality concepts that are irreducible so to speak.

I am sick of theory papers that adds human interpretation to connect a mathematical result to some fancy terms that everyone seeks after in real-world ML applications. Ironically, chatGPT is the most productive way to strip off those advertisement so the essence is better exposed. https://chatgpt.com/share/6ae78178-c2ec-83ea-9bc3-fd304c64ec...

how come all the examples in demo have wrong answers

https://huggingface.co/spaces/galilai-group/prh_demo

> A man in a black shirt and hat with a yellow neckerchief has his hand raised.

no hand raised

> A white sheep sitting on top of a cake.

no its just cake sheep . sheep is not sitting

> A man is standing on the beach while holding his surfboard.

Its not just a man

infact all the examples are wrong.

I thought the platonic representation hypothesis was much more ambitious: wasn't it supposed to apply for any model across modalities (e.g. including human minds), rather than just the same model but with different modalities? I'll have to actually read the paper but I suspect there to be very strong assumptions which defeat the interesting aspects of the platonic representation hypothesis.

I built a ppt with claude for investors with the exact same phrase "AI infrastructure for proving and improving ... " on Oct 27 and sent it to investors. I suggested the title during the chat and claude agreed a lot. It ends up as a title of this paper 2 days later.