There are techniques which already achieve great compression of the cache at 4 bit, eg using hadamard transforms. Going from 4 bit to 3 bit isn’t the great leap people expect this to be. It’s actually slower to run and is generally worse in practice.