logoalt Hacker News

rybosworld • today at 8:11 PM • 1 reply • view on HN

These nerf trackers do not work. I wish people would stop posting them.

They don't detect a quantized model until the quantization gets down to 4 bits or less.

Humans absolutely do detect quantization well before 4 bits...

Try it yourself if you want. Try a small local model, both quantized and not, and I would be very surprised if it isn't immediately obvious which is which.


Replies

j_maffe • today at 8:30 PM

But quantized local models also dip in evals.

➕ show 1 reply