How to Block AI Crawlers from Your Website (2025)

Which AI bots crawl your content, why you might want to block them, and the exact robots.txt code to do it.

Try the free tool

No signup. Runs in your browser. Takes 10 seconds.

Which AI Crawlers Are Visiting Your Site?

Since 2022, AI companies have deployed crawlers to collect training data from the web. Unlike Google's Googlebot (which helps your SEO), these crawlers consume bandwidth and content without providing ranking benefits in return. As of 2025, the major AI crawlers are:

Crawler	Company	Respects robots.txt?
GPTBot	OpenAI	Yes
Google-Extended	Google	Yes
CCBot	Common Crawl	Yes
anthropic-ai	Anthropic	Yes
FacebookBot	Meta	Yes
Applebot-Extended	Apple	Yes

The robots.txt Code to Block All AI Crawlers

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Keep Google and Bing crawling normally
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Should You Block AI Crawlers?

Block if: You create original content and don't want it used to train AI without compensation
Block if: You want to be listed in AI search features on your own terms
Don't block if: You want your content to appear in AI-generated answers (AI search visibility)
Don't block if: You're a news or information site that benefits from AI citation

💡 Use the "Block AI Crawlers" template in the ToolsCourt Robots.txt Generator to add all these rules automatically, then download your robots.txt file.

Ready to try it?

Free, instant, no signup required.

Open Robots.txt Generator Free →

← All guides