alt Hacker NewsI'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
I'm not fully understanding this business of MoE so please forgive me if this is a dumb question, but would it be possible to use MPI with a small cluster to distribute the load?
It’s a good question.
In theory MPI could distribute experts across nodes. In practice, for small clusters the added network latency usually hurts more than it helps.
Better suited for big clusters with fast interconnects. For now we're focusing on single-machine speed (caching, GPU hybrid, etc.).