alt Hacker NewsThis is cool. I definitely think the "micro" sized LLM space is underappreciated, so it's always good to see work like this. I foresee a paradigm in some contexts where you have a hierarchy of LLMs, with more competent models actively training smaller models to solve specific tasks very efficiently, and something like this could be the smallest layer in that stack.
With that being said, the web demo is not particularly impressive. It really doesn't like anything I throw at it. I'm fine with accepting that fine-tuning is the solution to this, but I wonder if there's anything to gain from a bigger model? I know it's completely counter to the whole point of this, but a 14MB binary using 28MB of RAM seems unnecessarily small and pretty arbitrary.
Like, what does a 28MB binary get you? Or a 140MB binary? Or a 1.4MB binary? I'm guessing the choice of 14MB came from minimizing the size as much as possible while meeting certain requirements/performance expectations, but even a Pi 5 has plenty more room to spare. Curious if there's a good explanation for this (which I may have missed in my skim of the post).
Replies
So, its not a general language model, focused on tool call strictly for tiny edge-devices. There are solutions everywhere for high-capacity devices, Needle is for sub-$200 devices.
I'm quite impressed by the results of the web demo, especially given its size and the precision with which it uses the three available tools (tested with German commands). I could imagine that this LLM would fit well into a setup with multiple micro-sized LLMs for different purposes; so 14 MB for precise tool invocation is a reasonable memory footprint when a number of other local models are running (e.g. STT, TTS + language models).
It's called a LM :). LLM stands for Large Language Model.