alt Hacker NewsAttention Is Not What You Need (2025)
Comments
Scale is all you need.
That's why you see other attention free large language models work. But in a world with constraints on time and compute the architecture does matter because scaling does have a cost (which is dropping)
lmao seems just computing power arms race >>> specific algorithm
(2025)
They only demonstrate this idea on a laughably small toy problem. An 18M parameter model (which they call “moderate”) on an ancient dataset proves exactly nothing. Plenty of bad ideas appear to work at trivial scale.
Arguably the most important aspect of transformers is that it allowed easy training of arbitrarily large models.
"subconscious" reasoning
Interesting idea, but it reads like an LLM write up. For me to engage properly with your written work, I need to see human effort. There are too many half-baked ideas out there, and unless you, as the author, thought the idea is worth putting proper effort into communicating, why should I think of this anything other than something someone said in passing? One way you could communicate effort to me, and likely others, rather than this work looking like the output of some LLM, is to use graphs, schematics and figures to support claims. Metrics are extremely misleading, only really useful when we've verified what is being measured is what one assumes is being measured. Metrics on their own are often deeply bullshit, so without any kind of graphs of trends or other important qualities, I assume it's all LLM output, since LLM's will default to not plotting anything while humans tend to go the opposite way.
Also, more effort into completeness needs to go in. Perhaps it is in there, but you just report an "accuracy" without defining what that means. The more I look, the more it starts becoming apparent this likely hasn't received any real human over-sight at all. "What does Grassmann mixing actually buy us?" It buys nothing, because it is a concept, Claude.
Neat idea, but in my opinion devoid of critical thinking / analysis or the level of rigour required to be suitable for broader dissemination. I'd put effort into making this a more credible body of work ASAP. Slop like this WILL cripple your standing in the field.
Interesting...
interesting
Reminder: Publications on arXiv are not peer-reviewed, and practically anyone can post a paper regardless of quality or correctness.
why doesn't the author have any internet footprint (or at any rate any one that links to this paper)? it's curious.
The author describes the model's pairing operation as:
(zt,zt+Delta) where Delta is a positive offset.
This means the model is using the next token to predict the next token. It should have been (zt,zt-Delta), or it's cheating.
I don't know if that's just the paper, or if the code was also b0rk3d.