AI Robots.txt Generator - Block GPTBot & AI Crawlers

Block AI training bots (GPTBot, ClaudeBot, Google-Extended) from your website. Protect content from LLM training while allowing search engines.

Quick Presets

Recommended: "Block Training, Allow Browsing" balances protection with visibility in AI answers.

AI Crawlers

Additional Options

Sitemap URL (optional)

Include explanatory comments

Group by category

robots.txt

Implementation Guide

Copy the generated robots.txt content above
Create/edit the file at your website root: https://yoursite.com/robots.txt
Merge with existing rules if you already have a robots.txt file
Test by visiting your robots.txt URL directly in a browser
Monitor your server logs for crawler activity (User-Agent headers)

Crawler Categories

Training Used to collect data for AI model training

Browsing Real-time web access for AI assistants

Dataset Builds public datasets used by multiple AI companies

Aggressive Known for high crawl rates or unclear policies

Nginx Config Generator - Server Configur…

XML Sitemap Generator - Create SEO Sitem…

Gitignore Generator - Templates for All …

HTAccess Generator - Apache Redirects & …

Robots.txt Generator - Control Search En…

View all Dev configurators →

Frequently Asked Questions

AI crawlers are bots that scrape website content to train large language models (LLMs) like ChatGPT, Claude, and Gemini. Unlike search engine crawlers that help people find your site, AI crawlers take your content to train commercial AI products without compensation. Blocking them protects your intellectual property.

GPTBot is OpenAI's crawler used for training future AI models. ChatGPT-User is used when ChatGPT's browsing feature accesses websites in real-time to answer user questions. Many site owners block GPTBot (training) but allow ChatGPT-User (browsing) to remain accessible in ChatGPT answers.

Google-Extended is a separate user-agent that controls whether your content is used for training Google's AI products like Bard/Gemini. Blocking it does NOT affect your Google Search rankings - that's controlled by Googlebot, which is separate. You can safely block Google-Extended while keeping Googlebot allowed.

No. Search engine crawlers (Googlebot, Bingbot) are completely separate from AI training crawlers. Blocking GPTBot, ClaudeBot, or Google-Extended has zero impact on your search rankings. These are distinct user-agents with different purposes.

Major AI companies (OpenAI, Anthropic, Google, Meta) have publicly committed to respecting robots.txt for their AI crawlers. However, smaller or less scrupulous scrapers may ignore it. Robots.txt is a request, not an enforcement mechanism. For stronger protection, consider also using meta tags or HTTP headers.

Common Crawl is a nonprofit that archives the web and provides datasets used by many AI companies for training. Blocking CCBot prevents your content from entering these public datasets. Many AI models, including some versions of GPT and Claude, were partially trained on Common Crawl data.

It depends on your goals. If you want to prevent AI training on your content, block training bots (GPTBot, Google-Extended, CCBot, etc.). If you still want your site mentioned in AI answers, allow browsing bots (ChatGPT-User, PerplexityBot). Use our presets to quickly configure common scenarios.

Bytespider is ByteDance's (TikTok's parent company) aggressive web crawler. It's known for high crawl rates and is suspected to be used for AI training. Many sites block it due to its aggressive behavior and unclear data usage policies.

New AI crawlers emerge regularly as companies launch new AI products. We update this tool's crawler list frequently. Major ones to watch: any new products from OpenAI, Anthropic, Google, Meta, Microsoft, Apple, and Amazon typically introduce new user-agents.

Image AI training typically uses different methods (often downloading images directly rather than crawling). To protect images, also consider: removing images from public access, adding watermarks, using hotlink protection, or serving images through authentication. The img2dataset tool is commonly used for AI image collection.