alt Hacker NewsRight. Nobody says it's necessarily economically the best option. But it's also an asset that has alternative applications and barely depreciates!
Right now you could extensively use an M5 Max for a year and sell it for almost exactly the same amount at the end. Maybe even more, at the current rate. And it can do other things for you: host your VMs, edit high def video, whatever. Even if it stops being useful to serve LLMs, it might be useful to run as a server for other tasks for a whole decade. It will be a fast, supported Mac for seven years. For the R9700, maybe gaming?
Depends what you want, doesn't it? Or are we all expected to conform?
For me personally, right now I have a secondhand M1 Max macbook pro, and that is my local AI machine, because my needs are exploratory and I want to run it here. I want to learn by comparing models. I want to learn how to run them and advise others how to (local inference is not a zero-sized market).
My next step up will be to take the apps I'm looking at building that run against these small models, and host those apps online with some sort of cloud inference — openrouter, digitalocean, cloudflare — for those same models.
If I ever find I want to vaguely describe some desktop application before bed and come down the next morning to an aggravating slop version of it written in Rust, I'll consider Claude.
For a local non-laptop, I've looked at the M5 Max and the R9700. I've also looked at the M5 Pro, which in practice is about as fast at token generation as my M1 Max (because the M1 Max's memory bandwidth is strictly theoretical), and smokes it at prefill. That alone might be enough for a "leave it running" mini AI box, and it will be a nice desktop machine. I just don't need it enough yet.
As an aside there's this thing — somewhat interesting, though constrained:
Uses the 5090 mobile chip, which is pretty good speedwise (about the same as the R9700) but 24GB. Small, tidy, on-prem. I think they have missed the boat on pricing, but if they can ship in volume, maybe not.
Great insights! And I'm kicking myself for not buying a 4090 3 years ago for $1600, shipped direct from nvidia... it seemed like too much to pay for a GPU at the time!