Pay to crawl is already here.

The usage patterns of how people pay and use AI is basically the same model the web should be using: you pay a small bit of money to access monetized pages, just how you pay a small bit of money to get AI responses.

It just needs people and browsers to get onboard with protocols. Crawlers will have no choice but to pay for content behind these 402 gateways.