> just crank it up when you get a flood,

A few months ago there was a story posted here about someone who completely eliminated crawlers on their website with Anubis.

I think it was getting upvoted before users were clicking the article because if you did, you had to leave the Anubis PoW page open for several minutes before you could get into the site. The Anubis difficulty scale is unintuitive and the difference between a small delay and becoming unusable is easy to cross.