alt Hacker NewsI use it as a drop in replacement for LLMs I had previously doing filtering and classification in a pipeline. It's loads cheaper and faster with roughly equivalent performance on my own internal tests. The confidence scores (which are also reasonably well calibrated on my data) are a super nice bonus.
Replies
vanuatu • yesterday at 10:04 PM
^ Same here. so much faster and cheaper than using an LLM to return structured data
For me it's using 1 Jev call to ask 5 questions, instead of what I had before.
What I had before was basically:
- 1 classifier question
- if ok, do some heavy computation
- after heavy computation, another classifier
So now what I do is:
- Jev with the 1st and 3rd classifier
- if ok, heavy computation
I can classify many things at once, and it's cheaper and faster than 1 for sure and 1 maybe classification that I had before.