alt Hacker NewsWhat kind of hardware are you using and what is your token generation speed (tok/s)? Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
What kind of hardware are you using and what is your token generation speed (tok/s)? Every time I've tried to run local models, even on my beefy mac book (128Gb Ram), I've been very disappointed at tok/s speed.
I've been using dual NVIDIA GPU's with Ollama. Even when I push to 30b models and more context window, Ollama manages RAM / VRAM very well. I never got OOM errors, just massive slowdown as the PCI-E bottleneck throttles the GPU's. I've had some large prompts take 30 minutes to process.
But I should mention: I am trying to implement workflows and processes akin to CI/CD that run 24/7 in the background. These are not interactive use cases, so I don't care so much about tokens per second.