I also no longer trust benchmarks on this one. When the context gets a bit longer and the problem harder low quant models often produce worse output for me. Sometimes they even loop.

Interestingly different formats also often behave differently. GGUF unsloth is so far the best for me.

QAT, which bitnet training is a form of, helps a ton in preserving accuracy at such low bits per parameter. There are also better quantization approaches that try to preserve the most sensitive weights† but are computationally expensive and so not typically done. Another complementary option is, if the model is fast enough, we should be able to push up correctness by self-consistency voting at close to T=1. Smart/fast Zero-shot classifiers like the recent Jev could help with aggregation across answers too, extending applicability.

†Every paper I've read estimates the average information content of transformer LLMs at about 3-4 bits per parameter. Curiously, biological synapses are also estimated to be about 4-5 bits per synapse, possibly a bit lower.

Well, you could train directly at this bitrate.

You are conflating post training quantization and low bit training.