I looked at their paper and the biggest gains come when you have many questions from one context, which is the way these models are supposed to be used. It's around 1.5-3x faster than vLLM with single questions. That's the worst case where you can't reuse anything.

The way I understand it most of their speedup comes from caching intermediate results. They freeze the state after processing context then branch each question off that point. It means they can process questions in parallel after the context stage is finished, and don't waste time doing the same work N times.

Thanks for the questions! I think tancop's summary has already captured the essence, and I can add some details.

> Under what specific circumstances is it actually 9.5x faster?

The 9.5x is Qwen3.5-0.8B on QuALITY, one long article that has 5.8k tokens and 18 questions on average. The more questions share a context, the bigger the gain.

> Are there any circumstances where it is just about as fast as something like vLLM? For instance, single user, single token/decision inference?

Yes, with a state that has only a single question, it ranges from a near tie with 27B on a long input to more than 3x with a small model on a short input. In this case nothing is shared, so the gain only comes from lower per-request overhead, since Flide runs the whole forward pass as one prerecorded CUDA graph.

> How did you achieve the biggest speed up?

The big speedup comes from cases where one state carries many questions, where we only compute the shared context once. We achieve this by using a mask that keeps the questions from seeing each other, and each question carries the correct position index just as if it came right after the state, so we can get the answers in one pass. The second is the prerecorded CUDA graph.

> Are there any tradeoffs? "While keeping exactness" implies not.

There is no accuracy tradeoff, since each question gets the identical computation as if asked alone. What remains is floating-point rounding that shifts slightly with batch size in every engine, even for a question asked alone, and our results are still the closest to the FP32 results among all the engines.

> Are there any specifics about how one must structure their context to take full advantage of it?

To get the most out of it, you can put everything shared in the state and ask all the questions in one request. And for continuous agentic decisions, use --session-cache-gb to get a further speedup.