You are viewing a single comment's thread from:
RE: Google's Bot Imposes a Large Impact on the Performance of SteemYY.com
I am not a robots.txt expert... but I think so...
at least you tell them which bots are allowed what pages and others are delayed crawling... or something like that
https://stackoverflow.com/questions/7787264/is-it-possible-to-control-the-crawl-speed-by-robots-txt