alt Hacker NewsWell they're quite explicit about that:
In summary, our results suggest that what we fundamentally need for strong sequence modeling is not attention as such, but a principled way for representations to move on the manifolds they inhabit. Grassmann flows provide one concrete instantiation of this idea, and we hope they will encourage further exploration of geometric alternatives to attention in the design of neural architectures.
They also note they kept the proposed model very simple to make it easier to understand (and presumably train):
In this paper we deliberately restricted ourselves to a minimal design—k = 2, no explicit global invariants—to keep the core idea clear. But we view “global invariants + local Grassmann flow” as a promising direction for future work on geometry-aware reasoning.
No my point is more general. The headline promises me that I don't need attention based transformers, then you see the benchmarks and they tell you they got 10% worse perplexity. How is that going to convince me that I don't need attention?
I mean if you look at e.g Mamba 3 [0] (see page 12 and 14 for benchmarks), the perplexity is roughly the same and the benchmark scores are roughly the same. If you tell me attention doesn't matter you have to convince me by showing that you can get the same performance with a variety of architectures. In short, I would have expected the paper to be a survey across model architectures that all perform roughly the same, which then would give us the impression "hey, they're all performing the same, maybe we don't need attention after all!", instead we get to be shown that their pet architecture is worse, which is the opposite of what the headline conveys.
If I gave a random ML researcher this paper, they would just walk away and work on the next transformer-mamba hybrid.
[0] https://arxiv.org/abs/2603.15569