alt Hacker NewsMinions is an internal enterprise system already in use at Apple. It pools idle Mac GPUs into a shared compute cluster for ML inference.
There is no distributed execution of a single LLM across multiple GPUs using tensor parallelism here. Instead, it relies on task-level parallelism-breaking down a large workload into independent subtasks that run across separate machines. For example, generating embeddings for a million documents.
Replies
greggsy • yesterday at 3:13 PM
My university ran its render farm across idle workstations using the screensaver back in early 2000. Same concept.
➕ show 3 replies
This pattern has been implemented across hundreds if not thousands of different organizations (and between organizations). I have a patent on data distribution strategies from my time at Google running a large (>1M cores 24/7 for 3 years) idle cycle harvesting project.
it's a good idea but you often end up with huge tail latencies, slight incompatibilities due to fleet heterogeneity, and a database/scheduler management problem.