>It's sole purpose is to deny the bots you don't want on your site.

This is fundamentally impossible.

In the end only pay per crawl will work. Like x402 but maybe something similar.

In other words: not only client has to spend some kind of resource for access, but it has to be a transaction, server must get something from serving the request.