Just make valid robots.txt and sitemap.xml, please, so I can crawl and update mirrors of the sites I am interested in with least amount of impact on the site.