logoalt Hacker News

written-beyondyesterday at 2:17 PM2 repliesview on HN

That reduces the quality of the response though.


Replies

debugnikyesterday at 2:45 PM

As opposed to emitting non-JSON tokens and having to throw away the answer?

show 2 replies
Der_Einzigeyesterday at 4:09 PM

THIS IS LIES: https://blog.dottxt.ai/say-what-you-mean.html

I will die on this hill and I have a bunch of other Arxiv links from better peer reviewed sources than yours to back my claim up (i.e. NeurIPS caliber papers with more citations than yours claiming it does harm the outputs)

Any actual impact of structured/constrained generation on the outputs is a SAMPLER problem, and you can fix what little impact may exist with things like https://arxiv.org/abs/2410.01103

Decoding is intentionally nerfed/kept to top_k/top_p by model providers because of a conspiracy against high temperature sampling: https://gist.github.com/Hellisotherpeople/71ba712f9f899adcb0...

show 2 replies