Why is Robots.txt Important?

Robots.txt is a file that tells search engine crawlers which pages or files the crawler can or cannot request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of a search engine's index.

Keep in mind that although it is expected that crawling bots will adhere to your requirements, and in fact, it is a strong etiquette signal to do so, they do not have to obey your robots.txt rules.

The role of the robots.txt file is important if you need to prevent your site from being overloaded from getting thousands of bot hits from different crawlers.

Do I have to block some crawlers?

Here is where you need to tread with care: if you block some spiders/crawlers, you may prevent your site from showing in the search engines results (SERP). If you inadvertently block the Google spider, you may potentially see a dramatic drop in your rankings, simply because Google cannot find your site.

A good strategy is by default allow all robots, but if you see in your logs that you are getting thousands of hits all the time from a bot you do not identity, you should talk the the person in charge of the technical SEO aspects of your website (that could be yourself) to discuss if it is perhaps a good idea to block that specific potentially malicious bot

All in all, visits of bots are often times good news. If you receive lots of visits from Google and other large search engines, that means that you are getting some good crawling budget.