alt Hacker NewsI don't know much about LLMs but does that mean you have N^2 computation with the context size since every token needs to track how it relates to every other token?
Replies
TomatoCo • yesterday at 8:16 PM
Yes, except no with the KV cache. Because tokens aren't modified by future tokens you can cache the meaning of previous tokens. This makes the total effort linear over the entire context (or constant per forward pass).
➕ show 2 replies
For full self attention yes