How to use the Robots.txt Generator
- Enter your sitemap URL.
- Pick a preset or list the folders to block, one per line.
- Choose whether to block AI training crawlers.
- Download robots.txt and upload it to your site's root folder (public_html).
Worked examples
| Rule | Meaning |
|---|---|
| User-agent: * | Applies to all crawlers |
| Disallow: /admin/ | Do not crawl /admin/ |
| Allow: / | Crawling everything else is allowed |
| Sitemap: https://…/sitemap.xml | Where to find your sitemap |
Frequently asked questions
Where do I upload robots.txt?
In the root folder of your domain, so it opens at https://yourdomain.com/robots.txt. On cPanel this is usually public_html.
Does robots.txt stop a page appearing in Google?
No. A blocked page can still be indexed if other sites link to it. Use a noindex tag and let Google crawl the page to see it.
Should I block CSS and JS files?
No. Google needs them to render your pages properly.
Does blocking GPTBot affect Google rankings?
No. GPTBot and CCBot are AI training crawlers, separate from Googlebot. Google-Extended controls Gemini training only, not search.
How do I test my robots.txt?
Use the robots.txt report in Google Search Console after uploading.
Written and tested by MasterKit Online Editorial Team
Our team builds and tests every tool by hand, checks each formula against official sources (IRS, HMRC, the Income Tax Department of India, CBSE, NHS, WHO) and reviews pages whenever rules change. Last reviewed on . Spotted a mistake? Tell us.