Unsloth Dynamic 3.0 GGUFs

It would be nice if unsloth published GGUFs would use a version number or something, because now I have multiple different files on local storage that otherwise have exactly the same name.

"Qwen3.8-27B-UD-Q8_K_XL.gguf" for instance.

The one downloaded at least 4 days ago is a different thing and is NOT the "Dynamic 3.0" GGUF which I am now downloading, which I presume will have a different sha256 checksum?

The unsloth page says dynamic 3.0 is released "today", but I have an older copy of qwen3.8 27B Q8 which I downloaded, if I remember right, at least 4-5 days ago...

https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

I mostly use local models when the data has personal information. Earlier this year, I felt the coding quality was still not as good as Claude Code.

One thing that works for me is to ask the local model to make some fake data with the same format, let Claude Code work on the fake data, and then bring the code back and run it locally on the real data.

This way the real data never leaves my machine, but I can still use a stronger model for most of the coding.

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.

These are very good!

I'm hoping for speed improvements because the only problem running the 27B model on my Macbook pro (M4 Max) is the speed: 20 tokens per second. I benchmarked and MTP actually makes things slower, so I disabled MTP altogether. I'm hoping there will be some breakthroughs or optimizations that will allow me to run this at 30-50 tokens per second, which would make a big difference.

Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.

I could of course download and test myself, but that would take days with my internet connection.

   "We also made some smaller UD-1bit quants with UD-IQ1_S being 6.2GB (without MTP) which retain around 72% top-1% accuracy yet being 89% smaller"

Since it seems like this not only improved sizes but also performance I can't wait for some benchmarks and comparisons. If you don't have a separate GPU for inference, every single GB matters so a comparison between specific Q4 Quants is really interesting to me.

Currently I very much can't decide between going for a bit of a lower Q4 Quant to squeeze out a bit of buffer and ctx or wondering if a slightly higher (IQ4_XS vs Q4_K_M/XL) is worth it

No Dynamic 3.0 NVFP4 quants just yet from the look of it, as a heads up. Would love to see how those perform relative to others on the curve.

Might be off-topic but: is it possible to perform such a quantization on Apple devices? Something like Mac Studio Ultra M1 (even if it would take weeks/months)?

Waiting for MLX version

The new IQ4XS has been working pretty well so far on 4090 16gb.

Is it possible to use a model that needs around 64 GB VRAM if you have four GPUs with 16 GB VRAM each?

It seems the NVFP4 quants have a preview version of this Unsloth Dynamic 3.0. Is this close to the finished version, or would it be better to switch to one of the newer quants?

huh, sounds like they’re talking about over fitting and datasets etc, it seems like this is almost more like a fine tune/distill than just a pure quantization

Sidebar: single threaded inference isn’t good enough anymore

Can this help tiny models like Qwen3.5-0.8B?

Would converting those quant to MLX preserve the accuracy/size ? Or this only work with GGUF?

No MLX versions for 3.8 though.

[dead]

[dead]

[flagged]

[dead]

Cool. Now run TerminalHard and compare to unquantized 27B.

KLD of 1%, or similar error metric that multiplies, on 10000 tokens would give accumulated error of 2,000,000%