alt Hacker NewsAn AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.
Replies
syntaxing • yesterday at 2:01 PM
Which quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.
That's about right. Half the TPS with half the power.
GLM-5.2 will be much more demanding tho