alt Hacker NewsA 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP.
Replies
pbgcp2026 • last Monday at 11:38 PM
[dead]
A 32gb card does run it nicely. I use unsloth's UD-Q5_K_XL at 256k context (k/v at q8_0), and get ~67 t/s on a 5090. I still need to look into MTP.
[dead]
Nice. I used Q4_K_M to have some headroom. But yours seems to fit nicely.