I sometimes still wonder exactly who these scrapers are

This has zero to do with AI (it's just a convenient scapegoat that fits the narrative). Absolutely everything to do with those pushing for centralised control of the Internet.