alt Hacker News> So, the whole idea here is that we increase the effective depth from 22 to 44 block applications without adding another set of transformer weights.
From what I gathered, LLM inference is bottlenecked on memory, right? Which implies there's "spare" compute we haven't been using? Does reusing the weights like this allow us to utilize it? (Do more math per unit of memory?)
Replies
the_real_cher • yesterday at 9:41 PM
I think it's bottlenecked on memory throughput. Someone else more knowledgeable can verify this.
You need to separate memory capacity and bandwidth. Looping decreases memory capacity/FLOP but not bytes loaded/FLOP, since weights need to be loaded again for the 2nd pass. Plus (depending on the method used) capacity required for KV will be that of the equivalent unlooped model (44 blocks) and KV is typically larger than weights at long context.