alt Hacker NewsGemma 30B with 256K context runs at 20 tok/sec on my M3 Max with 128GB RAM so I think there’s something wrong with your setup. This should run at ~30-40 toks. Maybe your inference engine is not optimised for Mac.
Replies
piyh • today at 4:40 AM
30B is useless on 24 gigs of ram as there's ~4 gigs of ram left for everything else even with unsloth quants
I just used their `Bonsai-demo` repo like this;
`cd ~/Code/Bonsai-demo && BONSAI_CTX=65536 ./scripts/start_llama_server.sh`
then used it in a very minimalistic pi with a very small system prompt.
Didn't spend much time to try to optimize it tbh, but my issue was not the speed. it just could not make a decision on how to implement the task, kept going on an on.