You'd need the 256 gb memory model which will be expensive because apple has trouble getting capacity (got turned down by cxmt). And even then you can only run a 2 bit quant which is noticeably worse than 8 bit

That’s not how that works. The hosted models don’t stay still in size and capability while Apple advances. Both will advance their frontier and there will still be a gap and developers will still prefer the stronger option.

Local vs remote compute is a constant thread in tech history - mainframes and desktops then local and cloud compute (think Google Photos bs Apple photos - one indexes on device the other indexes in cloud). Now we have the next chapter local vs cloud LLM models.

There will always be a market for frontier labs in the cloud based models - these models will always be able to be bigger, and that will likely translate to doing things local models can’t.

Logically also we’ll likely get to a point where RAM drops in price as production ramps up, and local LLM is both capable and cost effective. This feels like it is coming for Siri / Gemini / Alexa personal assistant type use cases.

So I think the local LLM will become a thing in laptops and phones in a year or two, offering PA type use cases. Professional LLM services will likely remain at the frontier (and in the cloud) for the foreseeable.

They will cost an insane amount as well. Maybe less than subscriptions or tokens. But running massive models on laptops with batteries and poor cooling doesn’t make much sense.

Every single MacBook built in the past half-decade already has an LLM built into the latest version of their OS.

But there's a significant difference in hardware required between running a 3B parameter model and a 700B-1T+ parameter model.

Sure buddy, all you'll end up with is a $10k machine that run gimped models at like 30tok/s for about 5m before the fan kicks in and it starts to sound like a turboprop, while offering maybe 30% of the context size of hosted models.

The RAM shortage situation won’t be sorted out within the next year.

I'm still waiting for Linux to topple Windows

> run free LLMs locally at native speed

This reads like a hallucination. What does native speed even mean?

I don't know why you'd want to burden your laptop with a large model. But I can totally see a new "developer workstation" product that's just a semi-large box that's optimized for running frontier open weights models for one to few users.

I'll give you credit for at least offering a specific, somewhat unique take. But this is a pretty dumb take lol

> Apple will release M7 MacBook Pros / Mac Minis next year

The latest on Apple is TSMC is stuck on the next iPhone due to lack of RAM. Good luck getting any Macs. Memory shortage is getting worse.