logoalt Hacker News

oofbey • today at 3:17 PM • 1 reply • view on HN

Sounds like they noticed a few extreme outliers were pulling the difference down and hacked it out. Methodologically that’s really sus to do this after the study is done. But generally I think it’s okay to exclude extreme outliers - we generally assume Gaussian distributions, but the real world happens and messes things up. Having a few kids that are 5 or 6 sigma out pulling your averages - I can believe the rules are different for them. But this should have been decided ahead of time.


Replies

rahimnathwani • today at 3:21 PM

I emailed the authors and received a reply. That part of the study was not preregistered.

My message to them:

  Dear Dr. Burland and Professor Kane,
  I was reading your November 2026 report, Phone-Free Schools and Achievement Growth, and saw that the study was preregistered on the Open Science Framework. I haven’t been able to locate the registration.
  Could you please share a link to the preregistration or let me know where I can find it? I’m particularly interested in checking whether the exclusion of extreme fall-to-spring score changes was specified in the original analysis plan.
  Thank you for your time and for making the research available.
  Best regards
Their reply:

  Rahim:  That part of the study was not pre-registered.   We have used the NWEA data from earlier years, but had not encountered such extreme outliers.  The magnitude of gains or losses we dropped were clearly errors—multiple standard deviation gains in a single year, also inconsistent with the midyear measurement.   I’m ccing our colleague, Samantha Cullum, who can provide the link to the pre-registration.   Tom Kane
➕ show 3 replies