alt Hacker NewsIf you load an MoE model that slightly exceeds your GPU memory budget, llama.cpp and ggml v1.0.0 can cache less-frequently used experts in host memory rather than failing to load.
If you load an MoE model that slightly exceeds your GPU memory budget, llama.cpp and ggml v1.0.0 can cache less-frequently used experts in host memory rather than failing to load.