Robots.txt Generator - Control Search Engine Crawlers

Create robots.txt files with visual builder. Presets for WordPress, Shopify, Next.js. Configure crawl rules, sitemaps, and crawl-delay for SEO.

Quick Presets

Add Rules

User-Agent

Path

Sitemap URL

Crawl-Delay (seconds)

Current Rules

No rules added yet

robots.txt

Syntax Reference

Directive	Description
`User-agent: *`	Applies to all crawlers
`Disallow: /path/`	Block this path
`Allow: /path/`	Explicitly allow (overrides Disallow)
`Sitemap: URL`	Location of XML sitemap
`Crawl-delay: N`	Wait N seconds between requests

Wildcard Patterns

* matches any sequence of characters

$ matches end of URL

Examples:

/*.pdf$ - blocks all PDFs
/private/* - blocks /private/ and subdirectories

AI Robots.txt Generator - Block GPTBot &…

Terraform Backend Generator - Configure …

HTAccess Generator - Apache Redirects & …

TSConfig Generator - TypeScript Configur…

Cron Expression Generator - Schedule Tas…

View all Dev configurators →

Frequently Asked Questions

Robots.txt is a text file at your website's root that tells search engine crawlers which pages they can or cannot access. It's part of the Robots Exclusion Protocol (REP) and is respected by major search engines like Google and Bing.

No, robots.txt only blocks crawling, not indexing. Pages can still appear in search results if linked elsewhere. Use noindex meta tags or X-Robots-Tag HTTP headers to actually prevent indexing.

Place robots.txt in your website's root directory so it's accessible at example.com/robots.txt. It must be at the exact root URL - subdirectory robots.txt files (like example.com/blog/robots.txt) are ignored.

Disallow blocks crawlers from accessing a path. Allow explicitly permits access (useful to override a broader Disallow). Example: Disallow /admin/ then Allow /admin/public/ lets crawlers access only /admin/public/.

Add 'Sitemap: https://example.com/sitemap.xml' at the end of your robots.txt. You can list multiple sitemaps. This helps search engines discover all your pages, especially new or updated content.

Crawl-delay tells crawlers to wait N seconds between requests. It reduces server load but slows indexing. Google ignores it; Bing respects it. Only use if your server struggles with crawler traffic.

Add a User-agent block for that bot: 'User-agent: GPTBot' followed by 'Disallow: /'. This blocks only that crawler while allowing others. See our AI Robots Generator for comprehensive AI crawler blocking.

Block /wp-admin/ (except admin-ajax.php), /wp-includes/, and search results pages (?s= and /search/). Don't block wp-content - that contains your images and themes that should be crawled.

No! Robots.txt is public and only a suggestion - malicious bots ignore it. Never rely on robots.txt for security. Use authentication, IP blocking, or .htaccess for actual access control.

Not required, but recommended. Without robots.txt, all bots can crawl everything. Even a simple 'Allow: /' file helps you control crawl budget, add sitemaps, and block unwanted bots in the future.