logoalt Hacker News

lukax • today at 9:02 AM • 1 reply • view on HN

I guess it's just capacity problems. They probably have a fixed amount of compute and they need to make space for new models that will be released. So they quantize existing models e.g. in half so that can use 50% of current capacity for new models. They have such massive scale that you just cannot dynamically get that amount of new compute online so they need to work with what they have. No provider has thousands of GPUs idle.


Replies

Rapzid • today at 10:55 AM

They don't do that. They've said as much and there is zero evidence to support it.

Under load requests queue and take longer.

➕ show 1 reply