The point of the post was that you need to add robots.txt (or similar) because it's a thing, and also tell us where they are.