alt Hacker NewsFound this benchmark for Jev-class models: https://benchmarkheaven.com/jev-models
There are already many Jev-like models in there.
Edit: No affiliation. Just found it and thought others might find it interesting.
Replies
raybb • today at 3:00 PM
One thing I'm still trying to figure out is how this compares to something like gliner. If you're just doing classification in what situations would you choose kev vs gliner?
The open source ones- I downloaded a number and tried them and compared to Jev.
Anything that required knowledge / familiarity mmBERT and ModernBERT post-trains performed much worse.
So it seems like they did some kind of useful expansive pre-training.
Things that were Qwen or Gemma Diffusion did better at those kinds of tasks but were generally pretty inconsistent in terms of whether they could succeed repeatedly (and be stable + reliable) on the many types of tasks that are in the cookbook part of the Jev docs.
If you ask Jev similar input + questions, it's pretty stable. And does a reasonable job on a lot of questions.
This one public benchmark (the only I've seen) seems to give the open versions way too much credit. It wasn't my experience at all.
It gave my a false wrong sense of what might be required to get it working for something at work to avoid needing a new subprocessor as - at least on Cloudflare / OpenRouter Jev is third-party not hosted.