robots.txt and the meta robots tag both talk to search crawlers, but they control different things. robots.txt manages crawling — which URLs bots may fetch. The meta robots tag (or X-Robots-Tag header) manages indexing — whether a fetched page may appear in results.
This distinction trips up even experienced SEOs. Blocking a page in robots.txt does not remove it from the index; it just stops crawling. If you want a page out of search, you often need to allow crawling so the crawler can see a noindex directive.
At a glance
Aspect
robots.txt
Meta Robots
Controls
Crawling (access to URLs)
Indexing (appearing in results)
Location
Single file at site root
Per-page tag or HTTP header
Blocks indexing?
No — only crawling
Yes, with noindex
Scope
Path patterns, whole directories
Individual pages
Best for
Saving crawl budget, blocking sections
Keeping specific pages out of search
When to use robots.txt
You want to stop bots crawling large low-value sections.
You're managing crawl budget on a big site.
You need to block access to entire directories.
When to use Meta Robots
You want a specific page kept out of the index.
You need per-page control like noindex or nofollow.
The page must still be crawlable so the directive is seen.
Verdict
Use robots.txt to steer crawling and meta robots to steer indexing — they're not interchangeable. The classic mistake is blocking a URL in robots.txt to hide it, which actually keeps a bare, description-less listing in results. To truly deindex, allow the crawl and serve a noindex tag. Test your robots.txt rules before shipping.
'/%3e%3ccircle%20cx='100'%20cy='100'%20r='84'%20fill='%23093e47'/%3e%3ccircle%20cx='100'%20cy='100'%20r='79'%20fill='url(%23teal)'/%3e%3cg%20transform='translate(55,55)'%20stroke='url(%23cmd)'%20stroke-width='11'%20fill='none'%20stroke-linecap='round'%20stroke-linejoin='round'%3e%3crect%20x='27'%20y='27'%20width='36'%20height='36'%20rx='5'/%3e%3ccircle%20cx='13.5'%20cy='13.5'%20r='13.5'/%3e%3ccircle%20cx='76.5'%20cy='13.5'%20r='13.5'/%3e%3ccircle%20cx='13.5'%20cy='76.5'%20r='13.5'/%3e%3ccircle%20cx='76.5'%20cy='76.5'%20r='13.5'/%3e%3c/g%3e%3c/svg%3e)