alt Hacker NewsThis is very interesting! Seems like a promising direction.
I wonder though if it supports the same claims as Jev: answers are not impacted by other answers to the same questions, nor the existance of other questions? It seems by sharing KV cache all questions will be visible. And I think the diffusion causes the answers to attend to eachother?
Also I think without fine-tuning we can not say that the probabilities it output are actually probabilities. Maybe fine tuning using Brier scoring would do the trick?
Maybe some kind of hierarchical structure of the KV cache can make questions independent, and with smaller diffusion canvas’ generated in batch can be a way to make answers generate independently?
Replies
> It seems by sharing KV cache all questions will be visible
Yeah, this is partially why in my approach I've done this instead, and not used diffusion model:
1. Convert the state into one shared prompt.
2. Run that shared prompt through the model once.
3. Fork the model’s internal state once per question.
4. Add a different question to each fork.
5. Ask each fork for its next-token scores.
6. Calculate only 64 possible label scores—not the whole vocabulary.
Basically ... skip decode.
Won't be as fast as doing diffusion model parallel across a pile of questions at once, but:
a) let's you use pretty much any existing text model (with some modifications). I've got qwen3.6 moe and qwen3.8 flash next running, am getting gemma4 working now
b) the problem you identified
It's possible I'm getting high on my own supply and misunderstand entirely the whole thing, but it seems to work?
https://github.com/rdaum/eider/commit/9c2d5c049068c33da2a48b...
I don't have the chutzpah to go creating PRs for vLLM to do the same.
I believe you get better answers by diffusing each question together, but the PR's server gives you finer control over that. If each question is independent, you can get better parallelism.