alt Hacker NewsThere is a lot of "AI demand" that isn't just running inference on an LLM whose weights you downloaded.
I'm training a model using reinforcement learning with self-play. I can and do use vast.ai when scaling but for experiments it's far faster, and cheaper, to run it locally until the bugs are all figured out. Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes. It's zero locally.
Replies
A C-level executive I know is getting a top-of-the-line new Mac simply to function as a personal build server and host for agentic coding instances - they are able to orchestrate so many parallel projects that they're hitting RAM limits from sessions and the builds and local test runs they're kicking off (largly unsupervised). Before AI, they'd only had a MacBook Air; this completely changes their workflows. They talk about how many other executives they've met are equally giddy at having gone from coding few to no projects themselves, to coding more projects in parallel than any of their respective pre-AI technical colleagues.
I'd suspect that agentic coding has birthed so many new effective engineers, that the entire dynamics of demand for high-end machines have been upended.
I’ve been happy training and running inference for small language models on my M4 Mac.
Inference with MLX is surprisingly zippy. I’m running a classification task on the entire HN comment dataset and it’s projected to take about two and a half days, which is not bad considering we’re talking about tens of millions of comments.
Yes, I could do it much more quickly by throwing Modal GPUs at it but this is low-priority work. I might as well throw my M4 a bone.
> Just provisioning a new instance and copying the relevant checkpoints and things can take 25 minutes.
Modal significantly improves this. Highly recommend.
Are you training something so big you need that much unified RAM though?
If you can fit it on a GPU, and especially for training, it is so much quicker than a Mac.
Same, but with vision models. Unfortunately, I might be at my limit locally. I have three models that I'm using to find and identify objects in pictures. The largest dataset and model now takes about 8 hours per epoch on my Mac M4 with 16G memory.
What are you training on using self play? Like alpha go? Curious what your setup is like .
I’m doing the same!
Do you find that CoreML manages to fill up your drive with so many tiny files that a reboot takes hours to clean them up? I keep meaning to get my friends still inside the spaceship to file a radar about that.
What game are you building?
Likewise. I have a huge demand personally to run AI noise-filtering models on many TB per month of raw video files. It takes about 3 days per file.
Apples ProRes codec is only licensed to run in high quality mode on a Mac, and so my Nvidia PC can’t do what I need. Thus, I own the beefiest Mac Studio you can currently buy. I would pay more for more TFlops.
I have done local LLM on there but it wasn’t interesting. Far worse performance and intelligence per dollar than the cloud boys.
There is no cloud offering for my video needs though.