logoalt Hacker News

vlovich123 • today at 4:16 AM • 2 replies • view on HN

Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?

Replies

solarkraft • today at 11:23 AM

Prefill is generally faster than generation, but not by much on older Mac processors. I get around 70-60 tps in prefill on my M1 Max for Muse Glimmer (not sure about the generation speed, probably between 15 and 30).

They allegedly improved this by “up to 7x” with M5 but I’m not sure about the exact numbers here.

kgeist • today at 4:20 AM

They mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-publ...

>device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate

I.e. low memory bandwidth.