alt Hacker NewsIf they are literally sending the entire transcript and asking for a summary, wouldn't almost all of the tokens be cache hit, and therefore fairly cheap?
Replies
buzer • today at 4:43 AM
Didn't they change the cache TTL to 5 minutes earlier this year? So depending on exact timing it could potentially be invalidated between the updates.
➕ show 2 replies
simoncion • today at 6:14 AM
A cheap operation adds up to a large cost if it's done enough times. TFA even talks about caching.
Do you disagree with the numbers and conclusion in the -pretty obviously bot-generated- Fine Article?
I don't want to sound rude but even at cache hit you are burning almost input * 0.10 * context amount of tokens multiple times.
If it happens 10 times you have already accepted the same cost as the entire session fresh prefill.
People on HN are either too rich or wayyy too rich