alt Hacker NewsStill need 2 or 3 days to properly quant this. But it is really good model to run if you have 64GiB or larger RAM on Mac. On M5 Ultra, you get ~3400 tok/s for prompt processing and ~50 tok/s for token generation, quite usable state.
Still need 2 or 3 days to properly quant this. But it is really good model to run if you have 64GiB or larger RAM on Mac. On M5 Ultra, you get ~3400 tok/s for prompt processing and ~50 tok/s for token generation, quite usable state.
Forgive the ignorance, but how are you running a 763B parameter model on 64GB of RAM?