logoalt Hacker News

no-name-here • today at 6:04 AM • 1 reply • view on HN

Cloud-hosted remains more efficient in terms of electricity, RAM use for large GB models, etc.

However, Apple is already including LLM models automatically with even iOS that can be queried offline for historical facts, etc.


Replies

oblio • today at 9:27 AM

I'm half optimistic about this, but we're so far from it from a hardware point of view.

We need unified memory for almost everything for these types of setups to be usable and we also need to standardize LLMs, if everyone wants to run their own LLM locally, that will probably be untenable for decades. We probably need new DRAM tech (3D RAM?).

Hardware prices need to go down a lot, and we're probably 5+ years away from that. Not only do prices need to go back to where they were, they need to become even lower. So we're probably 10+ years away from usable good enough local LLMs for the masses.

It's going to be an ugly decade from a hardware economics point of view. The potential is endless, but the way there is hard.