alt Hacker NewsThis tool has very tiny quants available for each model, and defaults to the tiny quants, so I'm not so sure you... tried it?
This tool has very tiny quants available for each model, and defaults to the tiny quants, so I'm not so sure you... tried it?
Ya but the library matters as well and what will get 20tps on one might get 30-40tps on another, especially with finicky cards like Intel arc.