alt Hacker NewsCould someone please share how such open source micro-LLMs might have been created?
Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?
Replies
HenryNdubuaku • yesterday at 10:19 PM
Technically, you could do that, but we trained this one from the ground up!
➕ show 1 reply
hgoel • yesterday at 11:32 PM
Another option for something this small and narrowly specialized could be to get traditional LLMs to synthesize the training data. Model collapse is probably less of an issue at this size relative to terabyte sized models.
There's a Manning book on creating your own LLM from scratch which answers your question exactly. There's another book from the same publisher specifically about small language models for specialty purposes.