alt Hacker NewsI'm working on something from a crappy laptop, those numbers from jev can totally be matched:
Local Latency: 0.1813 secondsReplies
cmrdporcupine • yesterday at 2:25 PM
and frankly for many of the kind of thing people probably want to use this for... you would want to run locally anyways.
why even bother with a network hop? build a specialized engine which does the prefill->measure cycle on local GPU with a model fine tuned for your application (e.g. gaming NPCs, autonomous driving, agricultural intelligence, drone.. target... selection, whatever)
I can also run a 0.6b model on my phone faster than openai can run astra, it doesn't mean my model is useful.