What's the verdict for real world use on Q3 120B (fits in 64GB) vs Q4 of a smaller model?
Bigger model wins as long as the quantization was done properly.
alt Hacker News
Bigger model wins as long as the quantization was done properly.