alt Hacker NewsI am sorry I did not understand all of it. But, would this allow running large MoE LLMs on a local network with experts spread out over multiple cheaper GPUs (or even CPUs)? This would perhaps be more useful than over the Internet, within offices for example.
That’s one of the strongest use-cases. On a local network (office, lab, home cluster) the RTT is a few milliseconds instead of 20-50 ms, so the expert-offloading becomes much more practical. You can spread the experts across several cheaper GPUs or even CPUs, keep only the dense parts + router on the machine you’re chatting from, and the whole thing stays private inside your LAN. No internet required, no cloud, just the machines you already have.