Why your local LLM feels dumber than it is

I just got qwen 3.8 27b mlx running on my Macbook Pro and honestly I’m pretty blown away by how not-dumb it is.

> And the comparisons in this post are not going to be running some 2.58-bit-gguf-in-ollama with a couple test prompts.

Genuine question : is there something fundamentally wrong with Ollama ?

I use Ollama because it is easy to set up and manage (and also because VLLM is not super Windows friendly).

I thought the main advantage of VLLM was better concurrency management (better batching).

But if the quality of the interference itself is an issue, then maybe I should reconsider my choice.

> I will make you read the really long unpleasant version with math.

This is the version I want to read :)

I assume it is unpleasant in spite of the math, not because of it?

most of the time when a local model feels dumb its not the quant, its the chat template. a lot of gguf mints just drop the template from the metadata and the runtime silently falls back to chatml. model still talks fine so nobody notices, it just gets noticeably dumber. got burned by this myself serving qwen, now i grep the gguf for the template tokens before i blame anything else. second place is sampling, people run whatever defaults their ui ships instead of what the vendor recommends and then compare that to benchmark numbers that were run greedy or with the official settings