alt Hacker NewsI'm one of the authors of this blog post. Figuring out how to integrate the LLM, the inference and serving engine, and the constrained decoding implementation such that you get the intended output and overall quality definitely requires careful engineering and testing. Please feel free to AMA. :)
Replies
ivanjermakov • yesterday at 1:58 PM
Do you know where <||> syntax is coming from? I find it very odd that existing markup wasn't used and there is very little info online about it.
➕ show 4 replies
febed • yesterday at 3:18 PM
Would this technique work with inference optimization techniques like DFlash and MTP? In other words does it slow down things to the point where its just better to retry?
➕ show 1 reply
thisandthatstud • today at 7:54 AM
[flagged]
So if I understand it correctly, the tool calls are not actually done by the model?
In the sense that the model manifests them, but they're not actually a result of the model doing anything but the inference engine filtering down the anything only to valid moves?
Could something like that also be used in other contexts?
I heard about how people dynamically steer models by massaging the weights a bit, but massaging the sampler(?) also sounds like that might give a lot of control over outputs with the weights staying technically the same?