Here is the fun part, the same companies invests millions of dollars to protect its infrastructure and services from scrapers/bots AND they also invest millions of dollars to circumvent the bots, captchas and rate limits.

Unfortunately you cant have it both ways which makes it hard problem to solve for everyone