alt Hacker NewsI wonder if NNs could be trained well enough with 1 bit weights (i.e. 0 and 1) with some layers doing addition, while others substraction (i.e. weight sign would be "hardcoded" into the network architecture). Or with zero-less weights (e.g. -1, -0.5, 0.5, 1).
There are 1-bit* LLMs (+1/-1, usually), though they often come with an additional scaling parameter:
https://huggingface.co/prism-ml/Bonsai-27B-gguf#weight-repre...
> Weight Representation: Q1_0_g128 Each weight is a single sign bit: 0 maps to −scale, 1 maps to +scale. Every group of 128 weights shares one FP16 scale factor. > Effective bits per weight: 1.125 (1 sign bit + 16-bit scale amortized over 128 weights)
The Bonsai ternary model also uses this group-wise scaling, so the theoretical bits per parameter is 1.71 instead of ~1.58; additionally, they actually store trits as 2 bits; with the group scaling factor it becomes 2.125 bits/weight: https://huggingface.co/prism-ml/Ternary-Bonsai-27B-gguf#memo...