alt Hacker NewsI've recently set up Ollama on my homelab server, which is a Topton N17 micro-ATX motherboard with a Ryzen 7 7840HS and 64 GiB of DDR5 RAM. It's no AMD Strix Halo, but it's a laptop CPU with a fairly beefy GPU and a decent amount of memory bandwidth.
I've had decent results with gemma4:26b (although it's more like batch processing when used as a coding agent) and my experience so far is globally in line with the first two sections of the ARCHITECTURE.md: fast with small contexts but performance degrades fast as its size increases. If the rest of the contents of that document are true, I'll need to retune my setup from scratch.
Replies
kennywinker • yesterday at 4:27 PM
Try strata. I’m not sure how well it handles igpus but if it does i’m pretty sure that system will go screaming fast
Try using llama.cpp directly and tune it for your hardware. I had gemma 26b a3b running at 400-500t/s prefill and 40ish t/s generation speed with a RTX A3000 GPU with only 4.5GB free VRAM. A better model than gemma itself may help you tube the parameters. Gemma wasn't that helpful in that respect.