LeCun and Balestriero say they prove the “platonic representation hypothesis” with JEPAs, and a 109M text model beats LLMs of up to 72B
In the paper “Proving and Improving the Platonic Representation Hypothesis with JEPAs”, Randall Balestriero and Yann LeCun claim to demonstrate mathematically why models trained on different data converge to similar representations of the world. On the practical side, a 109-million-parameter JEPA text model aligns better with the DINOv2 and CLIP image encoders than all the language models evaluated, of up to 72 billion. According to On The Wire, it is a preprint without peer review, and the proof holds in an idealized scenario.