logoalt Hacker News

julianlam • today at 1:33 AM • 0 replies • view on HN

Something doesn't seem right. Vanilla llama.cpp can hit 50 t/s on an R9700. Q6 (albeit with limited context of 64k, more if kv cache is quantized) fits entirely on the card.