Side note! Nginx was by no means the first web server to use a non-forking mechanism, nor the first open source web server to do so. Certainly Zeus (which was closed source) was earlier and very useful in this sort of application, and so was thttpd (open source, still exists as Merecat). I used thttpd quite a bit for single box applications and at one of my employers, nginx replaced a mixed strategy with Zeus, Apache and thttpd (and we tested one other whose name I can’t recall).

Non-forking httpd servers using select() were a popular little coding challenge for a while in the 90s. Spinner was one of them.

Nginx’s real strength was being able to proxy and cache HTTP using that same mechanism, so you didn’t additionally need to deploy Varnish or some other appliance.

As to whether this is a good mental model for what is coming for local LLMs, I am not sure I am convinced. Apart from more quantisation-aware training, perhaps binary and ternary aware training, custom inference engines per model, and maybe some improvements in diffusion models, the grand challenge in small footprint LLMs is training really small reasoning and tool use models, and so far it’s far from clear they can deliver.

Truly tiny models will not be viable as general coding assistants; even 12B dense is too small and you will find plenty of people who will tell you that 26B/4B or 35B/3B MoE is too. Though perhaps they can be trained for single languages, like just Python or just TS/JS.

More likely is the idea that 30-40B dense models might be good enough for most things once low cost and likely bespoke hardware catches up.

But I don’t think any truly profound advances seem likely in software or training alone. I am no expert but it feels like we’re already a lot closer to efficiency than we were in your analogy, and the gains are perhaps not going to be much more than small increments.

Maybe we will see something like a ternary 60B/10B MoE model turn up. But at the moment at least I am not sure where the incentives are to train these.

Everyone keeps repeating this who doesn’t understand the underlying technology.

Small llms are still way more efficiently server on big GPUs.

Sharing server capacity takes advantage of the massive parallel throughput and sharing of memory bandwidth.

You are sharing the GPUs with thousands of concurrent users.

I don’t remember that and I was there! The idea that the performance delta between Apache and nginx for any normal workload is anything like 20,000% is absurd.

What specific technical signals make you think we're close to a shift like that?

> Remember when we needed 200 servers for an enterprise website because Apache used one process or thread per connection

I don't, but holy moly. That sounds insane!

> And I think the data center buildout will end in carnage.

Isn’t it more likely that they will still all be used to the max? I don’t see how at any rate we will be going “alright, that’s enough intelligence for now”

> It’s going to move us from the big iron era of AI to small portable brains. Nature has already proved it’s possible with 20 watts and very little heat generation.

Agree. I know enough about the vagaries of scientific progress to not put any money on any timeline but directionally, that's where we are headed.

> And I think the data center buildout will end in carnage.

Disagree. And this is quite the leap from the previous statement, btw. The carnage happens if the demand for general purpose GPU compute disappears and even then there are so many ways to salvage the asset.

It'll need a change in architecture for that to happen. For example Geometric Reasoning that's being actively worked on by people like this:

https://www.sophontic.ai

> Nature has already proved it’s possible with 20 watts and very little heat generation

Never thought of it in those terms before.

First point is plausable, moving from bigger models to smaller models. But the nature thing is a bit of an overstatement, yes our brains are very efficient but they are fundamentally different from LLMs so it doesn't really map.

What does this have to do with Muse Glimmer 30B?

Nature takes its own sweet time to come up with photosynthesis or the krebs cycle. What takes 2 billion years for Nature to work out, these large systems will soon do it in 2. They have capacity to compress time in ways the chimp troupe cant.

20 watts huh? How much energy has nature sure is required for lift? Does the same amount of energy scale by size for airplanes?

Can these models compete with Cerebras inference performance? Why would I pay for a 2400 baud modem when DSL is available?

brains do it with 20 watts because theyre analog. llms require massive amounts of power and this isnt changing any time soon without a breakthrough