alt Hacker NewsNice! I would love to use it for images as well. Then again is using Grammar-Based Decoding with a json response not the same? Is Jev just that with nice caching? Because then I have been using that already…
Replies
allanrbo • today at 5:18 PM
Almost the same. The neat trick here is to ask the model to reply with just 1 letter, which is one token, rather than a long string of json. Its quicker. And just the fact that Jev made a pretty decent API for structuring your questions. And they do the massive parallelization.
ulam2 • today at 10:16 AM
Yes, that is my question too. Someone knowledgeable can comment
A functional difference is the probabilities associated with each option. True, you could use the constrained next-token distribution at the appropriate generation step, but those would not be calibrated by any means, which Jev's claim to be. An empirical analysis would be interesting.