alt Hacker NewsIf “speculative” approach works so well in different contexts why not make it first class and use everywhere, possibly recursively?
Replies
doctorpangloss • today at 5:10 PM
Multi-token prediction is a good enhancement to training. It isn't necessarily useful for inference. Other speculative decoding like EAGLE is. It is specific to the technology and the authors of these things write about it.
Speculation is only worth it if you can profit from it. Not every context allows this or has a similar idea of what can be speculated.