A robots.txt generator is a free online SEO tool that builds a valid robots.txt file for your website. Add user-agent groups, set allow and disallow path rules, include your sitemap URL and crawl-delay, then copy or download the finished file and upload it to your domain root.
Upload the finished file as robots.txt to your domain root (example.com/robots.txt).
Your robots.txt
What is a robots.txt file?
A robots.txt file is a plain-text file at your domain root (example.com/robots.txt) that tells search engine crawlers which parts of your site they may or must not crawl. It uses User-agent lines to target bots and Allow/Disallow rules to open or block paths. It is a crawl instruction, not access control: it does not hide pages from people.
Most sites need only a few lines: allow everything, disallow admin and staging areas, and point to the sitemap. Over-blocking is the common mistake - one stray Disallow: / can deindex an entire site.
How do I block a page from Google?
Add a Disallow rule for the path in robots.txt, but know this only stops crawling - Google may still index the URL if other sites link to it. To truly remove a page from search results, use a noindex meta tag instead. Use robots.txt Disallow for crawl-budget control (admin areas, filters, staging), and noindex for pages that must not appear in Google.
The two tools work best together: noindex removes the page from the index while robots.txt keeps crawlers from wasting budget on it.
What is crawl-delay in robots.txt?
Crawl-delay asks polite crawlers to wait a set number of seconds between requests, which protects small servers from aggressive bots. Google ignores crawl-delay, so manage Googlebot's crawl rate in Google Search Console instead. Use this field for Bing and smaller crawlers, and keep the value modest - very high delays can slow down legitimate indexing.
- Name the user-agent for each crawler group, or keep the * wildcard for all bots.
- Add Allow and Disallow path rules to each group.
- Enter your sitemap URL and an optional crawl-delay.
- Copy or download the file and upload it to your domain root.
Frequently asked questions
Is this robots.txt generator free?
Yes. Build and download unlimited robots.txt files with no account and no daily caps. Everything runs in your browser.
Where do I upload the robots.txt file?
Upload it to your domain root so it is reachable at example.com/robots.txt. Crawlers only check that one location.
Does robots.txt hide pages from people?
No. It only asks crawlers not to crawl paths; anyone can still open the URL directly, and the file itself is public. Use noindex for pages that must not appear in Google.
How do I block just one bot?
Add a user-agent group with that bot's name (for example GPTBot) and a Disallow: / rule. Other bots keep following the wildcard group.
Should I add my sitemap to robots.txt?
Yes. A Sitemap: line pointing to your sitemap.xml helps every crawler discover it without needing Search Console access.