alt Hacker News> 33M-parameter text JEPA reaches .125 on the PRH paper’s benchmark, against .034 on Flickr captions, level with Qwen2.5-3B and -7B. They need fewer samples.
Im bullish on jepa models. Success of autoregressive next token prediction feels like a stroke of luck.
Replies
NitpickLawyer • today at 8:46 AM
> Im bullish on jepa models.
Eh, dunno. They've shown lots of promise "in the lab", on these narrow domains shown in papers, but AFAIU they're hard to train on some domains (i.e. language). Someone would have released something if it's that promising. Even RWKV had more models trained that showed real comparisons. Jepa seems like a theoretical toy, until they actually prove that it works in real world (hah!) models...
> Success of autoregressive next token prediction feels like a stroke of luck.
Hard disagree. Self-attention + MLP was a godsend because it keeps scaling with compute ("the curve goes up and up and up"). Autoregressive next-token prediction turned out to be the most compute efficient way to train it.
We are almost 10 years into this, and it's still here (and in a field that moves at light speed!).
This is not a stroke of luck. It's here to stay.