alt Hacker NewsI was under the impression mixture of experts models allow, during the processing layer 37 for 1 token for 1 user, only a subset of layer 37's weights need be read from ram - dramatically reducing memory bandwidth, since weights are far bigger than activations.
However, if that machine is processing requests for 10k users right now, they'll all need different experts, and the machine will end up reading every weight anyway.
I was expecting big providers to split the work between machines, such that machine 1 processes experts 1-100, machine 2 does experts 101-200 etc. Each request would be sent to the specific machines for the experts needed, and results collected again ready for the next layer (on another set of machines).
However I don't see code for any of that.
Missed optimization, or am I wrong about the best way to slice the math into the hardware?
I wonder how quickly the latency between machines would add up. I'll admit this kind of process routing is interesting from an OR perspective.