alt Hacker NewsDepends on the workload. H100 will never have the network performance of Vera Rubin. There's also token per watt, newer systems will beat the older systems.
Depends on the workload. H100 will never have the network performance of Vera Rubin. There's also token per watt, newer systems will beat the older systems.
It's not clear how much of the latest chips have even made it on-line yet.
The claims of many GW of installed training/inference have come under scrutiny lately. The first VeraRubins aren't even there yet, so it's all GB300 NVL72s as the peak performers and probably <<1GW of those so far. Even xAI Colossus is mostly H200s and B200s.
Electricity costs are also a huge differentiator. When drawing 100kW the difference between >50cents and <10cents per kWh is pretty big! One is almost $0.5M and the other is less than $100k.