Would love to see these numbers reproduced on consumer GPUs, not just A100s.
This is an efficiency improvement that significantly lowers the amount of RAM you have to look at, on average, during decode.
It should improve performance on most hardware because most LLMs are memory bandwidth bound during decode.
Maybe somaday an 8gb videocard can be used for coding...
alt Hacker News
This is an efficiency improvement that significantly lowers the amount of RAM you have to look at, on average, during decode.
It should improve performance on most hardware because most LLMs are memory bandwidth bound during decode.