alt Hacker NewsLLMs are memory bandwidth bound not compute bound.
Replies
AntiUSAbah • today at 8:15 AM
LLMs are bound by both and depends on the hardware which factor is higher.
icelancer • today at 7:40 AM
This is only true for some parts of the time cost function.
This is incorrect, prompt processing is compute bound.