Robots.txt Generator

How to use the robots.txt generator

1. Under Start from, pick a preset: Allow everything, Block everything, Block AI crawlers or WordPress. You can then change every line.
2. In each group, list the User agents the rules apply to, one per line. * means every crawler that has no group of its own.
3. Add Allow and Disallow rules with Add a rule. Paths start with / and are case-sensitive.
4. Optionally set a Crawl-delay for crawlers that support it, and add more groups with Add a user-agent group.
5. List your Sitemap URLs, one absolute URL per line.
6. Review the Checks, then Download the file (it is already named robots.txt) or Copy it, and upload it to the root of your site, for example https://example.com/robots.txt.

How robots.txt works

A robots.txt file tells crawlers which parts of a site they may fetch. It follows the Robots Exclusion Protocol, standardized as RFC 9309 in 2022. The file only applies to the exact host and protocol where it is served: https://shop.example.com/robots.txt doesn’t cover https://example.com.

Each group starts with one or more User-agent lines followed by rules. A crawler uses the group whose user agent matches its name most specifically, and ignores the others; if nothing matches, it falls back to the * group. Within the group, Google applies the rule with the longest matching path, and when an Allow and a Disallow rule are equally long, Allow wins. That is why Disallow: /wp-admin/ combined with Allow: /wp-admin/admin-ajax.php works.

Two wildcards are supported by the major search engines: * matches any sequence of characters, and $ at the end of a path means “the URL ends here”. For example, Disallow: /*.pdf$ blocks every URL ending in .pdf. An empty Disallow: means nothing is blocked, which is how this tool writes a group with no rules.

Checks performed

• A group without a user agent, which crawlers would ignore.
• Paths that don’t start with /, contain spaces or accented characters (these must be percent-encoded) or use $ in the middle.
• The same path both allowed and disallowed in one group.
• A user agent that appears in several groups: Google merges them, but other crawlers may read only the first.
• Relative or invalid sitemap URLs, and a file larger than 500 KiB, the limit after which Google stops reading.
• A Crawl-delay aimed at Google, which ignores it. Bing and some other crawlers do respect it; for Google, the crawl rate adjusts automatically to how fast your server responds.

About AI crawlers

The Block AI crawlers preset keeps normal search engines allowed and adds a group that disallows the whole site for GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Google-Extended, Applebot-Extended, PerplexityBot, Bytespider and meta-externalagent. Google-Extended and Applebot-Extended are not separate crawlers: they are tokens that tell Google and Apple not to use your content to train or ground their AI models, without affecting normal search indexing. New AI crawlers appear regularly, so review the list from time to time.