vllm has limited support for quantized models, whereas ggufs are the native format for llama.cpp so it was always an easy choice. Also I never need to do batch GLM inference, whereas for e.g. photo captioning or embedding for semantic search, batch Qwen makes more sense.
35 tok/s for GLM 5.3 flash q4 quant
100 to 200 tok/s for Qwen3 VL 8b ; but it's usually short context, single image, captioning that kind of thing. It's generally sub-second for captioning a single image.
This is hybrid GPU/CPU inference for GLM (9995wx + RTX Pro 6000); and RTX Pro 6000 for Qwen3 VL.
I can't directly swap the setup because GLM 5.3 flash is huge, even q4 quant, it doesn't fit in 96GB. But it does fit in my 768GB of system RAM.
alt Hacker News
vllm has limited support for quantized models, whereas ggufs are the native format for llama.cpp so it was always an easy choice. Also I never need to do batch GLM inference, whereas for e.g. photo captioning or embedding for semantic search, batch Qwen makes more sense.
35 tok/s for GLM 5.3 flash q4 quant
100 to 200 tok/s for Qwen3 VL 8b ; but it's usually short context, single image, captioning that kind of thing. It's generally sub-second for captioning a single image.
This is hybrid GPU/CPU inference for GLM (9995wx + RTX Pro 6000); and RTX Pro 6000 for Qwen3 VL.
I can't directly swap the setup because GLM 5.3 flash is huge, even q4 quant, it doesn't fit in 96GB. But it does fit in my 768GB of system RAM.