logoalt Hacker News

throwa356262 • today at 5:32 AM • 2 replies • view on HN

Are 0.6b models useful without fine tuning?

Half of the times I ask qwen 0.6b "what is 1 + 2?" it ends up in a thinking loop of "but wait, the user is asking me to ..."

Replies

rhdunn • today at 3:41 PM

If you don't want the thinking, you can pass `enable_thinking: false` to the `chat_template_kwargs`. If using promptfoo, this can be done via:

    providers:
      - # llama-server
        id: openai:chat:qwen
        config:
          apiBaseUrl: http://localhost:7876
          apiKey: "..."
          passthrough:
            chat_template_kwargs:
              enable_thinking: false
The looping may be due to quantization -- I've seen it on locally quantized Q6_K Qwen 3.5/3.6 models. I recall seeing somewhere (here or r/LocalLlama) that Qwen models are sensitive to quantization of the keys, though I haven't yet experimented with/looked into fixing this. (I've been building up my promptfoo tests/infrastructure to detect looping, etc. on Qwen and other models.)
kamranjon • today at 6:54 AM

A fun thing I do with Qwen 3.5 0.8b is to take a screenshot of the Hackernews homepage and ask it to give me a JSON representation of the data and it does surprisingly well. With a well structured prompt I think it could be made to be pretty reliable tool for that type of task out of the box.

➕ show 1 reply