logoalt Hacker News

nlitened • yesterday at 7:40 PM • 5 replies • view on HN

As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks.

From what I usually see, ML researches just throw everything at the wall and see what sticks. Whatever approach turns out to work better, gets used more often, and sometimes theoretical background is developed for why it works. RoPE works not because it’s good math, just it turned out to work better than some other equally arbitrary approaches at some point, and then other approaches have also been developed that work better than RoPE.


Replies

Mabusto • yesterday at 9:29 PM

This is the most frustrating part of trying to understand machine learning. You can read a bunch of really great math books that rigorously build concepts from the ground up (calculus, linear algebra), then you start to get into the more hand wavey math (_some_ statistical concepts feel like this) and then you start getting into some of the theory behind machine learning systems and expecting the same amount of rigor and ability to understand _why_ something works and it’s always an unsatisfying explanation. I really wish there would be more disclosure like “this is what we tried and this seems to work, we think this is why” rather than present the method, give some mathematical formalization of the method, then go “it works great” and then move on. It leaves people like me wondering if there is something that I’m missing or some hole in my knowledge that is obvious to everyone else, OR does no one actually know why this works and just don’t want to say. Just give us a hint, make the shrug emoji or something, the more I learn about machine learning, the more I get that it really is more art than science.

➕ show 1 reply
fc417fc802 • yesterday at 9:09 PM

Both things can be true and useful. You can search for solutions in a fairly arbitrary manner, and when you find something that works you can attempt to develop a theory to explain it in the hope that said theory will enable more efficient search efforts going forward.

D-Machine • yesterday at 9:39 PM

This is exactly correct.

mathisfun123 • yesterday at 7:46 PM

> As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks

It's always like this because people can't admit ML is just spray and pray. So they superimpose theory.

➕ show 1 reply
esafak • yesterday at 11:30 PM

There is no enlightenment in this. Are you going to make the next breakthrough by throwing yet more mud at the wall, or are you going to take educated steps based on a "post-factum" analysis of what has worked well?