The other side of this is that we can still make software more efficient, and make better use of the old hardware than we had ever thought possible.

I’m doing more with a decade old GPU, which was manufactured before “Attention is all you need“, than I could 5 years ago, when quantization techniques were implemented.

I’m holding on to my 32 bit machines.

Most linux distributions dropped support for them (for good reason). But at the end of the day these machines are a fabric of up to ~ 4 billion bytes that can be used in a myriad of ways, and we only covered a fraction of the state space before we had moved on.