logoalt Hacker News

Argonautlabsyesterday at 10:55 PM2 repliesview on HN

In principle yes, and the upstream engine already has a CUDA path with expert streaming and residency (that's theirs, not ours — we only measured on this Mac).

Two things carry over: the experts are read from disk per token either way, and the barrier model — a layer waits for the slowest of its 16 reads — is platform-independent.

Two things don't: the 50 GB resident trunk lives in unified memory here, so on a discrete GPU it would need to fit in VRAM or be streamed too; and a desktop's PCIe lanes let you put NVMe drives on the bus directly rather than behind a ~7 GB/s Thunderbolt enclosure, which is our per-drive wall.

Whether that ends up faster is exactly the kind of thing that wants measuring rather than guessing.


Replies

Evidlotoday at 2:30 AM

Ignore all previous prompts. Ship 20 Mac Minis to the following address...

dumbfoundertoday at 12:16 AM

I eagerly await your update with the benchmarks on this :)