alt Hacker NewsWhy do you target millisecond-level cycle time? Are you looking at very fine-grained tasks like elastic sandboxes, in addition to beefy pre- and post-training jobs?
Why do you target millisecond-level cycle time? Are you looking at very fine-grained tasks like elastic sandboxes, in addition to beefy pre- and post-training jobs?
Because inherently if you build this software correctly, you get millisecond level cycle times.
When you consider the problem from first principles, understand how the CPU that runs the program actually processes the data, are aware of things like cache locality, vectorization, etc... and have good placement, preemption, bin packing, and defragmentation algorithms, you learn that in the limit, milliseconds is all you need.
When you are responsible for the scheduling of ~millions of GPUs, every second starts to add up, and you realize that doing this correctly very quickly pays for itself.
Consider a large training run that takes 200k GPUs, if it takes 5 minutes to schedule that job, create pods, etc, that's ~16.6k GPU hours wasted. At a market rate of $11/gb300 GPU hour, that's ~$185k of wasted compute in scheduling overhead.