alt Hacker NewsThey're not free to run, Kimi K3 needs to be run on the cloud, and the quantised versions aren't as capable. Unless you happen to have 3 - 5 TB of VRAM and an 8-node cluster of 8× NVIDIA H100s to run the full fat version. Plus the weights are not yet available to download in any case.
Replies
bmitc • yesterday at 4:21 PM
Qwen was mentioned in the comment I replied to and can run locally.
I agree that quantized versions aren't perfect, but using GLM5.2 as an example, the gap between a BF16 and something like a Q8-K-XL as published by unsloth or a similar Q8 quantization is very minimal. For other "large" LLMs there's a fair number of tests showing that Q8 is about 94% as good at literally half the size in GGUF files on disk, and half the RAM usage. Approx. 1500GB for the BF16 vs 820GB for Q8-K-XL.