alt Hacker NewsThat is a shift of comparisons though, right? This thread was about running locally, explicitly as an alternative to cloud services. I would guess few people have graphics cards with > 12GB ram.
Replies
wernam • yesterday at 8:56 PM
AMD Ryzen AI/Strix Halo is for running locally on your hardware. Openrouter is for evaluating the models you'll be running locally, before buying the hardware.
In any case, 128-256GB unified memory systems running MOE models is the "cheap" option, RTX6000 is the step above it.
➕ show 1 reply
A few more will have Macs with 48GB RAM though. (Indeed if you push it a little, 36GB unified is possibly enough for Muse Glimmer; I have not tested this theory)
There is, for sure, a limiting factor in the x86 architecture, but I think the RTX Spark will change that a bit, and maybe AMD's subsequent Ryzen AI/Strix Halo chips.