you can probably run Gemma4 26B on your card also at 4 bit. World of a difference compared with 12B.
Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?
alt Hacker News
Where does “big model highly quantized” start getting worse than “smaller model less quantized”? Is there a general formula or is it just trial and error?