How it works
- Name the site type: WordPress blog, Shopify store, SaaS marketing site, documentation site, or a custom setup.
- List any paths crawlers should skip, one per line. Leave it empty if you only want the defaults for your site type.
- Paste your sitemap URL if you have one. Without it the file simply has no Sitemap line.
- Click Generate Robots.txt, copy the code block, and upload it to the site root.
What a good robots.txt does
It keeps crawlers away from pages that waste crawl budget or leak private paths (admin screens, carts, internal search results, filtered duplicates) and points them at the sitemap. It does not protect secrets, since the file is public, and it does not remove pages from the index on its own.
Example
Site type "Shopify store", no extra paths, sitemap at https://example.com/sitemap.xml returns a file that disallows /cart, /checkout, /account, /orders and internal search, allows everything else, and ends with the Sitemap line. Each Disallow gets one sentence under the block explaining why it is there.
Tips
- Fewer rules are safer. Every Disallow is a chance to block something you wanted indexed.
- Use noindex for pages, robots.txt for sections. A single thin page needs a meta tag; a whole parameter space needs a rule.
- Check after CMS changes. Plugins and platform updates sometimes rewrite the file.
- Keep the sitemap current. The Sitemap line only helps if the file behind it lists live URLs.
Frequently asked questions
Does robots.txt stop a page from being indexed?
No. It stops crawling, and a blocked URL can still appear in results if other pages link to it. To keep a page out of the index, let it be crawled and add a noindex meta tag or header.
Should I block CSS and JavaScript?
No. Google renders pages, and it needs those files to see the page as a visitor does. Blocking them can hurt rankings and mobile evaluation.
Should I block AI crawlers like GPTBot?
Only if you do not want your content used or cited by AI assistants. ChatGPT, Perplexity and Google AI Overviews send referral traffic and citations to sites they can read; blocking them removes that source.
Where does the file go?
At the root of the domain, so it is served at https://yourdomain.com/robots.txt. One file per host; a subdomain needs its own.
How do I test it?
Search Console has a robots.txt report under Settings that shows the file Google fetched and any errors. Fetch the URL in a browser too; if it returns a 404, crawlers treat the site as fully allowed.
Related free tools
Try KoalaChat
Need more advanced AI capabilities? KoalaChat offers GPT-5, real-time data, and more, all with a generous free tier.
Try KoalaChatTry KoalaWriter
Need help with longer content? KoalaWriter helps you create blog posts, articles, and more with AI assistance.
Try KoalaWriter