llama.cpp supports a wide variety of 4-bit and smaller quants and mmap's models by default, so you dont need to be able to hold the weights in memory (the OS will handle bringing them in from storage as needed)

Its cool to see this implemented in a tiny amount of code without dependencies, but does it actually bring more performance?