They're already doing this under the name "fast answers"

[1]: https://www.reddit.com/r/OpenAI/comments/1stsxvc/new_feature...

Like google search, this does not work because of how common long tail use is.

What you think could be a big chunk, is more likely to be a fraction of a percent of queries.

And what use is similar query caching - so you (very often! if actually cost effective, maybe half the time) get a response to a query that was different from yours. Including for when you have a lot of context input already. You’re going to get trash.

If it were constrained to only very common initial prompts, and somehow the long tail did not actually dominate as it does with Google search (can't find the reference at the moment but it was a famous article some years ago), it also wouldn't account for serious enough cost savings. Long context is what is expensive.

This might only work in constrained domains like customer service where there’s tolerance for generic answers and escalation paths. For technical work? For general purpose use, with secretly canned responses charged at full price?

[dead]