related and possibly more general purpose https://github.com/t8/hypura
With so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...
alt Hacker News
With so many people implementing their own SSD streaming for specific combinations of model+hardware, maybe we should look into upstreaming to antirez/ds4 or llama.cpp...