logoalt Hacker News

alvatechtoday at 6:21 PM2 repliesview on HN

TIL that 1 bit models are actually 1.58 bit with three values +1, 0 and -1


Replies

NitpickLawyertoday at 6:43 PM

There's two variants of this (or, as the joke goes, for very big values of bit):

Ternary Bonsai 27B uses ternary {−1, 0, +1} weights with FP16 group-wise scaling, giving a true 1.71 effective bits per weight.

1-bit Bonsai 27B uses binary {−1, +1} weights with the same group-wise scaling, giving 1.125 effective bits per weight.

show 1 reply
bensyversontoday at 6:31 PM

Yeah, it's an unfortunate convention from the very first "1 bit" model. But to be clear, Bonsai comes in both ternary and actual 1-bit variants.