alt Hacker NewsI would say they do compound until proven otherwise.
Having "Wait, bar is not true, so that won't work" is not necessarily a correction. In fact, the problem is: across a long text it is a correction of a single mistake, but we are talking about thousands here.
But yes, of course that was a rough estimate. But the problem is - we don't really know what we are measuring here. Maybe there's a 2,000,000x difference of intelligence between coding indexes 52 and 50. By some measure that just feels small because that's how we process it akin to audio db.
Regardless the point is KLD and whatever they came up with is not meaningful. And they did not publish comparisons on real benchmarks.
Just at a sniff test level, don't you think that if the quantization resulted in anything like 2M% error in a pretty typical context length, it would be plain as day? You'd do an A/B test and one of them would look like standard generated text and one of them would veer into incoherence? If not - what on would 2M% error even mean then?
> Regardless the point is KLD and whatever they came up with is not meaningful.
I'm not saying you're wrong, I'm just saying this isn't a meaningful metric either, mostly because it is using a different type of error (divergence along a trajectory) than what was actually measured (divergence at a fixed point) and so can't be used for this purpose. It could establish an upper bound but going by your work that upper bound is so high it may as well be infinite. That's somewhat concerning but doesn't necessarily suggest it performs badly in a typical case (which is how I'd interpret an expectation of 2M% error).