alt Hacker NewsIs the hardware really getting better? It feels performance per watt is not getting better at all which is the metric that will matter eventually when supply-demand stabilizes.
As it is, it seems the improvements are about making the hardware cheaper (as in capex, not opex).
This is just feels from me from what I hear on the news and see on the products though.
Replies
Solar power and batteries are getting cheaper and cheaper at the moment. So Watts should become cheaper in the long run.
Especially when chips are becoming cheaper (in the capex sense), then you can afford to only run them when power is cheap.
Btw, from where do you take the notion that performance per Watt ain't increasing? We are also still using what's more or less general purpose GPU hardware; we could get a lot further if we were willing to specialise more. Which would be the natural avenue to explore, if progress in general purpose hardware slows down. Google is already looking.
The DGX Spark apparently consumes up to ~150W while being able to run many models at decent speeds.
I think really good efficiency is possible right now, but the GPU makers don't want to make their consumer GPUs too good for AI - if the cards were more efficient, it'd be much easier to run multiple - while data center ones have a bunch of additional power overheads.