logoalt Hacker News

LoganDark • today at 11:51 AM • 1 reply • view on HN

Optimize llama.cpp? Hmm.

WRT speculative decode, basically zero finetunes keep it. I'm testing with some ridiculous abliterated amalgamation so spec decode has been gone for most of its ancestry.

Fable recommended n-gram speculation so I'm working on that now.

Replies

mirekrusin • today at 2:18 PM

Optimize startup params for llama-server for your hardware (not llama.cpp itself), on my 2x 4090 I got ~20% speedup after maybe 40 mins.

ps. ngram didn't work for me very well, but dedicated speculative model works very well

ps. 2. in my case I'm just maintaining Makefile that does everything from update/upgrade (git pull/recompile) to starting server with different models, stuff like:

    # over baseline at temp 0.6 (95 vs 45 tok/s), ~4x over naive layer-split baseline.
    Qwen3.6-27B-MTP-UD-Q8_K_XL:
      ./llama.cpp/llama-server \
        -hf unsloth/Qwen3.6-27B-MTP-GGUF:UD-Q8_K_XL \
        --no-mmproj \
        --parallel 1 \
        --kv-unified \
        --flash-attn on \
        --fit off \
        --split-mode tensor \
        -ngl 99 \
        --cache-type-k q8_0 \
        --cache-type-v q8_0 \
        --host 0.0.0.0 \
        --tools all \
        --jinja \
        --ctx-size 262144 \
        --spec-type draft-mtp \
        --spec-draft-n-max 6 \
        --temp 0.6 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --repeat-penalty 1.0 \
        --presence-penalty 1.1 \
        --threads 8 \
        --reasoning-budget 2048 \
        --reasoning on \
        --chat-template-kwargs '{"preserve_thinking": true}' \
        --reasoning-budget-message "reasoning budget consumed, time to answer now"

    ...

    Qwen: Qwen3.6
    Qwen3.6: Qwen3.6-27B
    Qwen3.6-35B-A3B: Qwen3.6-35B-A3B-MTP
    Qwen3.6-35B-A3B-MTP: Qwen3.6-35B-A3B-MTP-UD-Q8_K_XL
    Qwen3.6-27B: Qwen3.6-27B-MTP
    Qwen3.6-27B-MTP: Qwen3.6-27B-MTP-UD-Q8_K_XL