alt Hacker NewsUnfortunately, I can't not use AI because it provides the best speech recognition I've ever used and, going along with hand problems, it now allows me to write code after a three-year absence from that skill. I do miss it. I'd never do it again professionally in the large, but I have missed writing code a lot.
My solution is to see what four or five-year-old equipment I can buy that will let me run local LLMs. I may only get six or seven tokens per second out of an i7, but it's a start. And best of all, I can turn the machine off when I'm not using it.
IMO, Migrating to small-scale local LLMs would be a significant improvement over using data centers.
Replies
> Migrating to small-scale local LLMs would be a significant improvement over using data centers.
This shift is probably inevitable, but it will vary significantly by region depending on prices for electricity. Look at, for example, the difference in fundamental homelab build recommendations between Germans and just about anyone else. Electricity prices in Germany are so high that even a now expensive Raspberry Pi or other ARM board is often preferred over Intel/AMD builds due to low power draw (especially low idle power draw), an effect that adds up for a machine running all the time over years.
With local LLMs and the GPUs to run it, especially if you want a model available to you all the time and can remote into your local network to use it whenever you want, there's no escaping much higher power draws, even at idle. Wherever electricity is expensive, the electric bill can be a prohibitive barrier.
> Unfortunately, I can't not use AI because it provides the best speech recognition I've ever used and, going along with hand problems, it now allows me to write code after a three-year absence from that skill.
I'll second this. The combination of Whisper + LLM makes speech recognition fantastic. I occasionally have arm pain from typing, and this is a Godsend.
I don't use it to write code - but in my experience stuff like emails + docs was the greater source of pain (one generally types slower while coding).
> IMO, Migrating to small-scale local LLMs would be a significant improvement over using data centers.
LLM serving is most efficient when you batch a lot of parallel requests together. Data center solutions also have the advantage of collecting queries from around the globe, so the hardware can be utilized around the clock.
Having everyone serve their own local LLMs would produce a lot more memory demand. Not less. The same memory would be idle most of the time, and when it was used it would be used for 1 person instead of a batch of requests.
There are other reasons to run local LLMs, but solving hardware demand problems is not one of them.