alt Hacker News> A recent analysis by the Dark Energy Survey Supernova Program also finds a preference for evolving dark energy, in the same direction as DESI DR2, but with a slightly lower statistical significance (bringing the combined significance down from 4.2 sigma to only 3.2).
Is this right? It would seem that adding another finding "in the same direction" should always increase the statistical significance, even if only slightly if the new evidence is week.
Under what circumstances could additional evidence for X reduce the estimated likelihood of X?
It is not true that another finding in the same direction should always increase the statistical significance.
Suppose you've got a coin; null hypothesis is that it comes up heads and tails equally often. I flip the coin 5 times and get heads every time. Probability of at least this many heads on the null hypothesis is 1/32. Now I flip the coin another 9 times and get 5 heads / 4 tails: evidence in the same direction. Between the two experiments I have 10 heads / 4 tails. Probability of at least this many heads on the null hypothesis is [(14 choose 0) + ... + (14 choose 4)] / 2^14 ~= 0.09, much bigger than 1/32.
There are also other circumstances in which even additional strong evidence for X can reduce the probability of X.
Suppose I have three hypotheses A,B,C which initially I think are all equally likely. Then two things happen that both have probability 1/2 if A is true, probability 1/4 if B is true, and probability 0 if C is true. After one of them, I should think A is true with probability 2/3. After both, I should be more confident, right?
Nope. Suppose e.g. what I'm doing is pulling balls out of a bag. Hypothesis A is "I have either a bag of red balls or a bag of blue balls, with equal probability". Hypothesis B is "I have a bag with 25% red balls, 25% blue balls, and 50% green balls". Hypothesis C is "I have a bag containing only green balls".
So I pull out a ball from the bag and it's red. That happens half the time in scenario A, 1/4 the time in scenario B, and never in scenario C, like I claimed.
I put the ball back and shake things up so I'm starting afresh, and pull out another ball. This one's blue. Again: half the time in scenario A, 1/4 the time in scenario B, never in scenario C.
But those two things can't ever both happen in scenario A, because in that scenario I have a monochromatic bag. They can both happen in scenario B. And of course neither of them can happen in scenario C.
So I got a result that (on its own) was evidence for A over the other two hypotheses, and then another result that (on its own) was evidence for A over the other two hypotheses, and the effect of both together is that I know A is false and B is true.