Question 1

What is robots.txt and what does it do?

Accepted Answer

Robots.txt is a plain text file at the root of your website (example.com/robots.txt) that tells search engine crawlers which pages or sections they can and cannot access. It uses simple directives like User-agent, Disallow, and Allow to control crawl behaviour. While it's not a security measure (it's only advisory), it's essential for managing crawl budget and preventing indexing of private sections.

Question 2

Does Googlebot respect Crawl-delay?

Accepted Answer

No. Googlebot ignores the Crawl-delay directive. Google manages crawl rate through Google Search Console's crawl rate settings instead. However, other bots like Bingbot and Yandex do respect Crawl-delay. If you need to control Google's crawl rate, use the 'Crawl rate' setting in Search Console.

Question 3

Should I include my sitemap URL in robots.txt?

Accepted Answer

Yes. Adding a Sitemap directive pointing to your XML sitemap helps search engines discover it. Place it at the end of your robots.txt file: Sitemap: https://example.com/sitemap.xml. You can include multiple Sitemap directives if you have several sitemaps.

Question 4

What happens if I have no robots.txt file?

Accepted Answer

If your site has no robots.txt file (returns 404), search engines assume they can crawl everything. This is fine for most sites. However, creating a robots.txt file is recommended so you can block access to admin areas, staging content, and other pages you don't want indexed.

Question 5

Can robots.txt block a page from appearing in Google?

Accepted Answer

Not exactly. Disallowing a page in robots.txt prevents Google from crawling it, but if other pages link to it, Google may still index the URL (showing it in results without a snippet). To fully prevent indexing, use a noindex meta tag or X-Robots-Tag header instead. Robots.txt controls crawling, not indexing.

Validate your robots.txt
line by line.

Three steps to clean directives.

Paste your robots.txt

Review the analysis

Test specific paths

Common questions about robots.txt.

Go beyond robots.txt.

Validate your robots.txtline by line.