For anyone curious about what the Gated Delta Network is: https://arxiv.org/pdf/2412.06464
Also, Gated Attention: https://arxiv.org/abs/2505.06708
alt Hacker News
Also, Gated Attention: https://arxiv.org/abs/2505.06708