What is robots.txt?

robots.txt is a plain-text file placed at the root of a domain — at example.com/robots.txt — that communicates crawling rules to automated bots. Following the Robots Exclusion Protocol, it lists user-agent groups and the paths each is allowed or disallowed from fetching. Well-behaved crawlers, including major search engines, read this file before crawling and respect its directives.

The file's syntax revolves around a few directives: User-agent names which bot a block applies to (with an asterisk for all bots), Disallow blocks access to a path, and Allow carves out exceptions within a disallowed path. Many sites also include a Sitemap directive pointing crawlers to their XML sitemap, helping search engines discover URLs efficiently.

A crucial and frequently misunderstood point is that robots.txt controls crawling, not indexing. Disallowing a URL stops compliant crawlers from fetching it, but if that URL is linked elsewhere, it can still appear in search results without a description. To keep a page out of the index, use a noindex meta tag or header on a page that is crawlable — not a robots.txt block.

robots.txt is also the tool for managing crawl budget on large sites, steering crawlers away from low-value areas like faceted search results, internal search pages, or admin sections so they spend their crawl allocation on pages that matter. Because a single wrong Disallow line can hide an entire site from search, testing rules before deploying is essential.