logoalt Hacker News

mathisfun123 • yesterday at 7:46 PM • 1 reply • view on HN

> As I understand, this is post-factum rationalization of what positional embedding is, not the actual theory used to build neural networks

It's always like this because people can't admit ML is just spray and pray. So they superimpose theory.


Replies

programjames • yesterday at 8:29 PM

Most people who publish papers in ML are alchemists, but please do not assume all of us are. The original authors of the RoPE paper knowingly designed their embedding to be translation invariant. I do not know if they were aware it was a group representation, but what they were trying to design was in essence a group representation.