logoalt Hacker News

HighGoldstein • today at 10:54 AM • 2 replies • view on HN

What GPU can you buy that will run a 180B model for $1800?


Replies

Zambyte • today at 1:01 PM

Qwen 3.8 Flash Next is 125B with 6B active, which with quantizing and system memory offloading (strata) runs nicely on 32 GB cards (I have actually run it on as low as a 16 GB GPU and 16 GB of system ram, but I don't recommend it for serious work). I was running it on a single r9700, but now I'm using 2 r9700s because I'm splitting my server with a friend, and they bought one as well.

On the single r9700 I was able to run q2 with 4 slots of 256k, and I think it was about 40 - 50 GB of system memory offloading? Got about 60 - 80 TPS per slot depending on concurrent use. With the two r9700s we are running q3 with 4 slots of 256k, using about 100 GB of system memory offloading, with mtp enabled. Getting about 70 - 100 TPS now.

SXX • today at 11:11 AM

You asking wrong question. You need to ask what tks/seconds it's gomna run at.

There are plenty of ways to run ds flash and qwen flash on single cheap gpu, but speeds are quite bad or contex windows are small.