Robots.txt Templates — Ready-to-Use robots.txt Examples

Choose from ready-to-use robots.txt templates: allow all, block AI crawlers, WordPress, e-commerce, and more.

Add rules to block GPTBot, ClaudeBot, Google-Extended & more
📄 robots.txt
# WordPress robots.txt

User-agent: *
Allow: /
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /feed/
Disallow: /trackback/
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search/

Related Articles

SEO

robots.txt Complete Guide: Syntax, AI Crawler Blocking & Common Mistakes

A practical guide to robots.txt — from basic User-agent and Disallow s…

Read more →

About Robots.txt Templates — Ready-to-Use robots.txt Examples

Ready-to-use robots.txt templates for WordPress, Shopify, Next.js, Laravel, e-commerce, and blog sites. Each template follows platform-specific best practices. Toggle AI crawler blocking (GPTBot, ClaudeBot, Google-Extended) with one click. Add your Sitemap URL, then copy or download the file. Free, no sign-up.

How to Use

  1. 1Select your platform — WordPress, Shopify, Next.js, Laravel, E-Commerce, or Blog.
  2. 2Toggle the AI Crawler Block switch to add rules blocking GPTBot, ClaudeBot, Google-Extended, and 4 more AI bots.
  3. 3The template updates in real time. Review the generated robots.txt content.
  4. 4Optionally enter your Sitemap URL to have it appended automatically.
  5. 5Click "Copy" or "Download robots.txt" when ready. For advanced customization, use the robots.txt Generator.

Features

  • Platform-specific robots.txt templates ready to copy in one click
  • AI crawler blocking toggle adds rules for GPTBot, ClaudeBot, Google-Extended & more
  • No robots.txt knowledge needed — templates follow best practices for each platform
  • Covers WordPress, Shopify, Next.js, Laravel, e-commerce, and blog/media sites
  • Free, no sign-up. Download the file and upload directly to your server.
01

Platform-Specific robots.txt Best Practices

Each platform has different default URL structures and paths that should be blocked. Using a platform-specific template ensures you cover all common paths without missing important ones.

WordPress: Controlling /wp-admin/ and /wp-includes/

WordPress sites should block /wp-admin/ (admin panel) but allow /wp-admin/admin-ajax.php since plugins rely on it. Never block /wp-includes/ CSS and JS files as Googlebot needs them to render pages. Also consider blocking /feed/, /trackback/, and search result pages (?s=) to save crawl budget.

Shopify: Blocking /cart and /checkout

Shopify sites benefit from blocking /cart, /checkout, /account (user area), and collection sort/filter URLs that create duplicate content. Shopify manages robots.txt through a robots.txt.liquid template in your theme code editor. Changes apply immediately after saving.

Next.js/Astro: Placing robots.txt in public/

For static site generators, place robots.txt in the public/ (or static/) directory so it is served at the domain root after build. Block API routes (/api/) and framework-internal paths (/_next/) to prevent unnecessary crawling. While some frameworks offer programmatic generation, a manually placed template file is more reliable.

02

When to Block AI Crawlers

Whether to block AI training crawlers depends on your site's content type and business model. Here's a framework for making the decision.

When You Should Block

Block AI crawlers if you produce original content (articles, research, photography, designs) and want to protect its value, if AI-generated summaries could reduce traffic to your content, or if you offer paid/subscription content. Blocking prevents your content from being used as AI training data without compensation.

When Blocking Is Unnecessary

For tool sites, SaaS landing pages, and e-commerce product pages, the impact of AI training is minimal. Being mentioned in AI responses can actually increase brand awareness. Even without blocking, always include a Sitemap directive and optimize your crawl budget with proper Disallow rules.

FAQ

Which platforms have templates?
Templates are available for WordPress, Shopify, Next.js/Astro/Nuxt (static sites), Laravel/Rails (frameworks), e-commerce sites (with cart/checkout blocking), and blog/media sites. Each is tailored to the platform's URL structure and common paths.
Can I use these templates as-is?
Yes. Each template follows widely accepted best practices for its platform. For sites with custom private paths, you can further customize using the robots.txt Generator.
Which AI crawlers does the toggle block?
The AI block toggle adds Disallow: / rules for GPTBot (OpenAI), ChatGPT-User (OpenAI), ClaudeBot (Anthropic), Google-Extended, Bytespider (ByteDance), PerplexityBot, and CCBot (Common Crawl) — the 7 most significant AI training crawlers.
How do I customize a template further?
Use the robots.txt Generator for full customization including individual path selection, per-bot AI blocking, Crawl-delay settings, and SEO bot blocking. The Generator gives you complete control over every directive.
Should I add a Sitemap URL?
Yes, strongly recommended. Enter your sitemap URL and it will be appended as a Sitemap: directive. This helps Google and other search engines discover your content faster.

Found a bug or something not working as expected?

Report a bug →