logoalt Hacker News

steno132yesterday at 8:35 PM6 repliesview on HN

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them.

I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.


Replies

adam_arthuryesterday at 8:36 PM

The entire universe of automation projects that can be run effectively for free relative to SoTA models?

I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models < 100B parameters.

Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running frontier models, only additional cost.

Gemma 4 26B outperforms most 100-200B models that I've tested for reasoning and structured output.

Gemma 4 12B can consistently select where to click on browser images given a minimal prompt, and do so very quickly.

show 2 replies
sowbugtoday at 4:37 PM

Whatever you're doing, try doing 500 or 1,000 of it in a batch. You'll exhaust any subscription quota you have, or if you're paying per token, you will probably find it too expensive. That's when you'll start to ask "how smart a model do I really need for this job?", and you'll investigate running a small but sufficiently capable model on your own PC, churning overnight through your 1,000 tasks.

mikeocoolyesterday at 8:42 PM

> I've been using Claude and GPT models for years

All 3 years?

show 1 reply
Zambyteyesterday at 8:36 PM

I like using my computer.

show 1 reply
user2722yesterday at 8:36 PM

There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.

show 1 reply
mannanjyesterday at 8:47 PM

I don't like the gaslighting of paying Anthropic or Open(Closed)AI and it being said its unsustainable for them to take my payment while simultaneously they take my data (edit: which is incredibly valuable) and I cannot opt out of that.

The obsession is for leaving hostile and abusive entities, the corporations or the people who fund them that have a horrible track record in regards to ethicality, rights and respect & human dignity.

show 1 reply