alt Hacker NewsHi HN! I'm one of the Flide team.
We built Flide from scratch to make Jev-like decision model inference up to 9.5x faster than vLLM and SGLang, while keeping exactness.
Here is the source code: https://github.com/OmniJev/Flide
Look forward to hearing your feedback and ideas!
Under what specific circumstances is it actually 9.5x faster?
Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?
How did you achieve the biggest speed up?
Are there any tradeoffs? “While keeping exactness” implies not.
Are there any specifics about how one must structure their context to take full advantage of it?