I hadn't looked at that study you selected, but yeah the methodology conflicts with the abstract (Also it low-key seems to be an ad for "Flexi 2.0.") It does seems to be a shady paper, with a small N and in a journal of questionable repute.

That said, there are 80+ other studies listed in the meta-study, which is pretty frank about its limitations. (Note the snippets about positive biases in the conclusion.) It is going more for quantity over quality and is transparent about the statistical findings of each one (or lack thereof; see the count of "Not reported"s.) All these references have a myriad of results, but across the spectrum of well-designed studies at reputable venues to the other end, they follow the same themes, so I don't think this can be dismissed that easily.

But if you want, here's more research (some of which I linked in a sibling comment https://news.ycombinator.com/item?id=48241839) which has similar findings:

https://scale.stanford.edu/ai/repository/ai-meets-classroom-...

https://arxiv.org/html/2601.20245v2 (from Anthropic)

This article summarizes some of the above and more studies and has similar findings: https://maxmynter.substack.com/p/learn-to-code-with-llms-i-r...