There are use cases where even low accuracy could be useful. I can't predict future products, but here are two that are already in place today:

- On the keyboard on iphones some sort of tiny language model suggest what it thinks are the most likely follow up words when writing. You only have to pick a suggested next word if it matches what you were planning on typing.

- Speculative decoding is a technique which utilized smaller models to speed up the inference for bigger models.

I'm sure smart people will invent other future use cases too.