alt Hacker NewsShow HN: Flide: Make Jev-like decision models up to 9.5x faster
Decision models need their own inference engine.
So we developed Flide, a high-performance and exact inference engine for Jev-like decision models.
Flide achieves up to a 9.5x speedup over vLLM and SGLang.
It now supports 30+ popular local Jev-like decision models.
Start to make your decision immediately:
pip install flide
Comments
Cool!
You really don't need an entire new inference engine for this. I modified vllm to work just fine for my decision model experiments.
It properly uses the single shared context and only recomputes the questions, or if the questions are longer, I let the questions go first and then the context that changes often can go after, and then the answer menu tokens... I got roughly the same speed increase with the right workloads that you're saying though.
I wanted to try this - essentially for my dataset I have a number of decision questions for the same input. I haven't figured out yet which local model to try first with it!
Hi HN! I'm one of the Flide team.
We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.
Here is the source code: https://github.com/OmniJev/Flide
Look forward to hearing your feedback and ideas!
do i need to host the model to do this ? if yes what model can i host in 3050 card
[flagged]
[flagged]
Flide is extremely fast when one input carries many questions, as the shared input is computed only once. Even with a single question, it can still be up to 3x faster, depending on the model size and input length.