logoalt Hacker News

civvvyesterday at 8:00 PM1 replyview on HN

I’m running Qwen3.8-27B-Unleashed UD-Q3_K_XL, which is a ~12.3 GiB Q3 quant, fully offloaded to the 16 GB 9070 XT. I disabled the vision projector to save VRAM and use one inference slot, Flash Attention, Q4 KV cache, --fit off, and --ctx-checkpoints 0. I’m running it with a 64K context window. The AMD driver also needs to be recent enough for ROCm 7.14; I targeted Adrenalin 26.6.4 or newer.


Replies

7speteryesterday at 8:10 PM

You can offload the vision projector to CPU/sysRAM

show 1 reply