The native weights are 4-bit for the sparse experts, and they quantize to ~80GB with limited degradation in real-world performance. That's a viable target for 64GB with SSD streaming, though it will be slower than keeping the whole thing in RAM (especially on a M2-class machine with its slower storage).
alt Hacker News
The native weights are 4-bit for the sparse experts, and they quantize to ~80GB with limited degradation in real-world performance. That's a viable target for 64GB with SSD streaming, though it will be slower than keeping the whole thing in RAM (especially on a M2-class machine with its slower storage).