Robots.txt Generator
Generate a clean robots.txt file for your website in seconds. Block crawlers from admin, staging, or private paths — and point search engines to your sitemap. Free, no sign-up.
Generate the link, copy it, and use it anywhere — website buttons, email signatures, social bios, or QR codes.
Configure your robots.txt
Select options below. The file generates instantly.
Your XML sitemap URL. Helps search engines discover all your pages.
Add any other paths you want to block from crawlers.
Adds a Crawl-delay directive. Note: Googlebot ignores this — use Google Search Console crawl rate settings instead.
Your robots.txt file
Copy and upload to your website root directory.
- Save the file as
robots.txt(no subdirectory) - Upload to your web server root — it must be at
yourdomain.com/robots.txt - Test it in Google Search Console → Settings → robots.txt
- Re-submit your sitemap in GSC after updating robots.txt
Common questions about robots.txt
Robots.txt is a text file in your website root that tells search engine crawlers which pages or paths they are allowed or not allowed to visit.
Not necessarily. Blocking in robots.txt prevents Google from crawling the page, but the URL can still appear in search results if other sites link to it. To fully remove a URL from search, use a noindex meta tag.
Yes. You should always disallow paths like /admin, /studio, /account, and /wp-admin to prevent crawlers from indexing internal management interfaces.
Indirectly. Properly configured robots.txt ensures Google focuses crawl budget on your important pages, not admin pages, duplicate content, or staging URLs.
Place robots.txt in the root directory of your website — accessible at yourdomain.com/robots.txt. It must be at the root, not in a subdirectory.
Yes. Each User-agent block targets a specific bot. Use User-agent: * to target all crawlers, or a specific bot name like Googlebot to target Google only.
The Sitemap directive tells search engines where to find your XML sitemap, which helps them discover all your pages faster.
Use Google Search Console's robots.txt tester under Settings, or fetch your robots.txt URL directly in a browser to confirm it is live and correctly formatted.
Robots.txt is one piece — get the full audit.
We check robots.txt, sitemap, canonical URLs, indexing, structured data, meta tags, page speed, and lead paths — and tell you what to fix first.