2026 SEO Standard · AI Protection Built-in

Robots.txt for WordPress

Don't use a 10-year-old generator. Protect your content from AI scrapers (GPTBot, CCBot, Claude) while ensuring Googlebot and Bing index you perfectly.

AI Scraper Protection

Instantly block 24+ known AI training bots (GPTBot, CCBot, Anthropic, etc.) from scraping your site for model training.

Platform Presets

Disallowed Paths

Sitemap URL

Generated robots.txt

User-agent: *

Disallow:
ℹ️ Your robots.txt looks healthy.

Why you need a "Modern" Robots.txt in 2026

Traditional robots.txt generators focus only on search engines like Google and Bing. However, today's biggest threat to website owners is IP theft via AI scraping. Bots like GPTBot (OpenAI) and CCBot (Common Crawl) crawl your site to train massive LLMs without permission.

🔍

Search Visibility

Tell Google exactly which parts of your site to index and which to ignore.

🛡️

AI Protection

Block training bots from using your unique content to train AI models.

🚀

Crawl Budget

Prevent bots from wasting resources on non-essential pages like admin panels.

AI Bot Blocklist 2026

Our generator includes a pre-vetted list of modern AI scrapers. When you enable the "AI Protection" toggle, we automatically add directives for:

  • GPTBot & ChatGPT-User (OpenAI)
  • ClaudeBot & Anthropic (Anthropic AI)
  • Google-Extended (Google AI Training)
  • CCBot (Common Crawl)
  • FacebookBot & MetaExternalAgent (Meta)
  • Bytespider (ByteDance/TikTok)

Frequently Asked Questions

Will blocking GPTBot hurt my search rankings?
No. GPTBot is OpenAI's web crawler used only for training their AI models. It is not used for search indexing. Blocking it will not impact your visibility on Google or Bing.
What is 'Google-Extended'?
Google-Extended is a user-agent that allows you to opt-out of having your content used to train Google's Gemini models. It does not affect Google Search indexing.
Where should I upload the robots.txt file?
The robots.txt file must be placed in the root directory of your website. For example: `https://yourdomain.com/robots.txt`.

Pro Tips for Technical SEO

  • Crawl Budget: Use robots.txt to prevent bots from wasting time on admin pages or search result parameters.
  • Sitemaps: Always include your full sitemap URL at the bottom of your robots.txt to help bots find new content faster.
  • Wildcards: Use the `*` wildcard carefully to block entire folder structures (e.g., `/temp/*`).
  • AI Protection: Regularly update your blocklist as new AI training agencies launch their own scrapers.

How to Use This Tool

  1. Select the 'WordPress' preset from our generator tool.
  2. Verify your WordPress installation's admin path (defaults to /wp-admin/).
  3. Enable 'AI Protection' to block training bots like GPTBot.
  4. Copy the resulting code and upload it to your WordPress root directory via FTP or a plugin like Yoast SEO.

What Is Robots.txt for WordPress?

WordPress is a powerful CMS, but its open-source nature means its directory structure is well-known. A properly configured robots.txt file is crucial to exclude non-public directories like /wp-admin/ and /wp-includes/ while allowing search engines to index your content.

Our WordPress preset automatically includes these exclusions plus specialized AI-bot protection to safeguard your content from unauthorized training models.

Why You Need This

Why is a custom robots.txt essential for WordPress sites?