Robots.txt Generator
Generate crawler and sitemap directives.
robots.txt sits at your domain root and tells crawlers which paths they may request. This builds one from your allow and disallow rules and sitemap URLs. Note that it controls crawling, not indexing.
Your input
Private, browser-based processingResults
Ready when you are
Enter your details above and run the tool.
How Robots.txt Generator works
What it does
Generate crawler and sitemap directives.
What you get
A robots.txt draft with crawler rules and optional sitemap URLs.
On your device
Your input is processed in this browser. Copy or download the result when you are ready.
What if a check cannot finish?
Confirm your input and try again. A timeout, blocked request, or unavailable provider is not proof that a domain is broken. Internet measurements and third-party services have limits.
Robots.txt Generator — frequently asked questions
Does Disallow keep a page out of Google?
No — this is the most common misunderstanding. Disallow stops crawling, but a blocked URL can still be indexed from external links. To keep a page out of the index, allow crawling and use a noindex meta tag.
Where must robots.txt be placed?
At the domain root, exactly at /robots.txt. A file in a subdirectory is ignored, and every subdomain needs its own.
Should I block AI crawlers?
Your call. Blocking them removes your content from AI-generated answers, which is increasingly a traffic source. Many sites now allow them deliberately for that visibility.