alt Hacker NewsI only see a 1-bit quant posted on unsloth HF and it’s 72.5 GB. Is that what you mean? That’s much bigger than I expected. If you can’t run a 4 bit quant in on Strix Halo it becomes a lot less interesting. https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF
Replies
Wheen • yesterday at 8:29 PM
Just a hunch, but it might be because of the 51B parameter n-gram embedding. At 125B, you'd expect ~16gigs for a 1-bit quant. Add 51gigs for the n-grams and you're not far off the actual size.
If that's true, it'd scale linearly with number of bits in the quant with an offset of about 51gigs. So Q4 should be a bit bigger than 82gigs, I'd guess in the 90s (as opposed to a ~280gig q4 if the whole 70gigs of the 1-bit quant scaled linearly).
In their page they say it will need at least 112GB[0], so including context, that would be a tight fit. I'm also hoping I can make a q4 fit on my 128GB strix halo
[0]: https://unsloth.ai/docs/models/qwen3.8-next#qwen3.8-flash-ne...