The tool has two modes. Create robots.txt builds the file from a template and your own rules and lets you copy or download it. Check robots.txt loads the live file of any site, or takes pasted text, finds errors and shows whether a given crawler may open a given address.
What robots.txt does and what it does not do
robots.txt sits in the root of a site and tells search engine crawlers which sections not to crawl. It is a convention, not protection: the file does not hide pages from people and does not remove them from search results. Google can still list a blocked page without a description if links point to it. To keep a page out of the index you need a robots meta tag with noindex, and to close a test site you need a password. Our .htaccess generator writes the password rules for you.
Templates
- WordPress blocks /wp-admin/ but keeps admin-ajax.php open, because themes and plugins depend on it, and hides internal search pages.
- OpenCart blocks the admin area, cart, checkout, compare and the sort and limit parameters that create duplicate pages.
- Block everything suits a test copy, but only together with a password.
- Block AI training crawlers adds a separate group for GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended. It does not affect normal Google Search.
How Google reads the rules
For each crawler the most specific User-agent group is chosen; if there is none, the group for * applies. Inside the group the longest rule that matches the address wins, and for equal length Allow beats Disallow. The * sign matches any sequence and $ marks the end of the address. The URL test in check mode works exactly this way, so you can see in advance whether important pages are blocked by mistake.
Common mistakes
- Disallow: / for all crawlers, left over after moving a site from its test copy.
- A relative address in Sitemap: it must be a full https:// URL.
- noindex inside robots.txt: Google ignores that line.
- Blocked CSS and JS files: the search engine cannot see the page the way visitors do.
- A file larger than 500 KiB: Google reads only the beginning.
What a WordPress site needs from its server is covered in our WordPress hosting guide, and the redirect checker shows whether the file is served with status 200 and no extra redirects.
Checking a live file
Check mode loads https://domain/robots.txt from our server, follows up to five redirects and refuses private addresses. A 404 means there is no file and crawlers may fetch everything, which is fine for most sites. A 5xx error is more serious: Google may pause crawling the site for a while. The number of checks is limited and file contents are not stored.
Upload the file to the site root with a file manager or FTP. On RichHost hosting you can also use SSH, and WordPress installs in one click; details are on our hosting plans page.
FAQ
Where should robots.txt be?
Only in the root of the host: https://example.com/robots.txt. Nobody reads it in a subfolder, and every subdomain needs its own file.
Do I need Crawl-delay?
Google ignores it, while Bing and some other crawlers respect it. If the server cannot handle the load, fixing the cause is better than slowing crawlers down.
How do I keep a site out of search while it is being built?
Protect it with Basic Auth. Disallow: / only asks crawlers not to crawl and does not guarantee the addresses stay out of search.
Can I block only AI crawlers?
Yes, switch on the matching option. Well behaved bots follow the rules, but robots.txt cannot force them technically.