> How should they know that your bot belongs to a potential customer and not to a competitor surveilling their prices?

That's the thing; they can't, really.

Best we can do is compromise; allow bots and rate limit them. Maybe someday there will be a relatively universal standard for fetching pages akin to the `index.html/index.md` pattern that is emerging now. Put the bot filters on the .html path and some sort of micro transactions system on the .md endpoint.

In basically every cat/mouse game defense is forever one step behind...