Here's a nice write-up on group theory and positional encodings: https://blog.janestreet.com/using-group-theory-to-explore-po...
This is better than TFA, thank you!
This is better than TFA, thank you!