alt Hacker News> Pangram boasts a false positive rate of 1 in a 10,000. That is, if Pangram says a block of text is AI there is only a one in ten thousand chance that it was written by a human.
That'd be if they had a false discovery rate of 1/10,000.
If for instance:
* 100,000 samples are tested
* 100 of which are AI-generated, the rest human-written
* Pangram flags 50 of the AI-generated samples (true positives)
* Pangram also flags 10 human-written samples (false positives)
Then the FPR is 1 in 10,000, but the chance that a flagged sample isn't actually AI (FDR) is 1 in 6.
Replies
In your scenario,
False negatives = 50
False negative rate = FN / (TP + FN) = 50 / (50 + 50) = 50%
Populare Science is built on a foundation of misunderstanding p-value.
https://www.graphpad.com/guides/prism/latest/statistics/comm...
(OP isn't exactly talkigh about p-value, because it's not comparing means, but the main mathematical structure of the Bayesian error is the same)
Of course, it's unclear if the article is using the wrong term, or the wrong definition, or if Pangram is just lying in the first place.
The classic. If there is one concept people should learn from stats it should be base rates and the Bayes update. It's very difficult to teach though. The mind just jumps to that interpretation, and even if you teach it, people just go right back to interpreting it like that anyway, because they think "yeah yeah technically something something math, I know, but basically in practice it kinda just means that anyway". Even plenty of educated professionals in science do not understand this and continue to get it wrong each day.