🧭
Crawling is not indexing
A Disallow rule limits crawling; it does not reliably remove a URL from search results. Use a noindex directive on an accessible page when removal is intended.
🔒
Not access control
Robots.txt is public and voluntary. Never use it to protect admin pages, private files or secrets; secure those resources on the server.
🗺️
Absolute sitemap URLs
Sitemap directives should use complete HTTPS URLs. Multiple sitemap indexes or language-specific sitemaps are supported.
How to use robots.txt safely
Upload the downloaded file to your website root so it is available at https://example.com/robots.txt. Rules are grouped by user agent and matching behavior can vary by crawler.
Test before blocking
A single Disallow: / can stop compliant crawlers from accessing the entire site.
Use root-relative paths
Rule paths normally begin with /. The * wildcard and terminal $ are understood by major search crawlers.
Do not block CSS or JavaScript
Search engines often need page assets to render and evaluate layouts correctly.
Validate after deployment
Open the public robots.txt URL, verify HTTP 200 and test important URLs in your search-engine webmaster tools.