alt Hacker NewsNice one! Tried it and unfortunatelly there are no small models that can fit my 16GB RAM or GTX1650 4GB GPU. Yeah, I know that this configuration is not meant to be used for AI/LLM work, but it would be good to provide support for some smaller models so that us plebeians can also play a bit with what you techbros are used to ;-) There are many small/very small models out there and I'm sure you could add a couple just for playing around and experimenting.
We have a couple bugs we are patching where we are reserving too much memory overhead, you should actually be able to fit a couple different models on there!
Plus in the near future, we'll add ways to automatically utilize your RAM as well (such as expert-streaming).
Our goal is to make the best use of whatever hardware you have, even if its not high end!