logoalt Hacker News

Sha1rholderyesterday at 11:34 PM2 repliesview on HN

> The MoE architecture makes a huge difference for being able to run these local models on reasonable consumer hardware

That's not true. For computers without unified memory architecture (which is the vast majority) VRAM capacity is the bottleneck for local models. In that case a dense model can deliver significantly more intelligence than an MoE model of the same size. And for a typical consumer/gamer Nvidia GPU, dense models are fast enough.


Replies

dannywtoday at 7:52 AM

Expert offloading significantly helps with the VRAM capacity.

Most MoE architectures have a few experts that are always running; this, the router, KV, and whatever else you have space for can stay in fast VRAM; and the remaining experts can be offloaded.

slimtoday at 12:58 AM

llama.cpp can run MoE with some layers in vram and some layers in ram