You are viewing a single comment's thread from:
RE: Google's Bot Imposes a Large Impact on the Performance of SteemYY.com
I had the same issue!!! google bot and also Bing bot impacted my site.
I changed the robots.txt and limited the pages they can crawl and it seems to be working... not 100% because some bots ignore robots.txt
For me, I don't want to ban entirely the Bot, is there a way to rate limit bots in robots.txt?
I am not a robots.txt expert... but I think so...
at least you tell them which bots are allowed what pages and others are delayed crawling... or something like that
https://stackoverflow.com/questions/7787264/is-it-possible-to-control-the-crawl-speed-by-robots-txt