alt Hacker NewsNot quite: not all of this needs to be in VRAM
It has a set of n-gram tables which you can stream from system RAM or even NVMe
That said it’s still quite big! I can’t fit it on my DGX Spark, though I believe you can if you have two?
Replies
rsolva • yesterday at 8:58 PM
I have access to two and will explore this the coming weeks.
➕ show 1 reply
It needs 3-4 Sparks to run well (at an acceptable quantization and sufficient KV cache):
https://github.com/christopherowen/spark-ds41f