alt Hacker NewsIf I had the capital I’d make an household inference appliance.
No peripherals except Ethernet, integrated compute (cpu+gpu+mem) and secondary storage (+mobo, psu). No accoutrements, just the minimum amount of hardware to run a model as a utility.
Even the appliance faceplate would be a display showing stats like an old HiFi stereo.
Edit: something like a series of modules consisting of a RISC-V CPU + Vortex GPGPU + memory
Replies
You're describing the mac mini/studio with some facelift.
I'm keeping an eye on Tenstorrent for this. Pricing seems like its going to end up being in between a super memory dense unified memory platform, and a purpose built GPU.
Definitely on the edge of what would make sense at home, but its interesting.
Isn't that what what George Hotz is doing over at tiny? https://tinycorp.myshopify.com/
I think the closest to that in existence is the LLM ASIC designed by Taalas:
Unfortunately their chatbot, while amazingly fast, doesn't know anything about the company running it.
Anyway I wouldn't mind an ASIC running a diffusion language model locally. Even if eventually it would become dated. Beats outsourcing all that to a company that's running on VC money which in the future might either perish or worse - dominate the market and charge whatever they wish.
Sounds like reinventing the home server.
the pheriphels support, or the appliance faceplate is tens of dollars, that not where you make the saving
95% of the price is going to be in GPU+CPU+RAM
build a Xeon / epyc 4u server. 12 channel ram.
[flagged]
It could heat your home in the winter and your pool in the summer.