Hm. I wonder if having a differentiable CPU in fixed weights might be useful-- you could propagate gradients through it. At inference time it could be replaced with a conventional implementation.

I think the RNN sidebar does the reader a disservice. Ordinary casual attention can do exactly the same thing-- it's just less computationally efficient (state grows with the number of bits vs the number of accesses) and so not as useful for qwenputer compiler.