logoalt Hacker News

kerenskiy • yesterday at 12:57 PM • 0 replies • view on HN

The context window fills up also/mostly from prefil which is supposed to be 20+ times faster or so. I think 40-80 tps for decode/output is believable for claude models