Robots.txt Generator — Create & Customize robots.txt

When a search engine bot visits your site, it first checks for robots.txt at your domain root. If the file exists, the bot reads the rules and adjusts its crawl behavior accordingly. This happens before any other page is fetched. A missing or misconfigured robots.txt can waste your crawl budget on unimportant pages, slow down indexing of your key content, or — in worst cases — accidentally block your entire site from search engines.

These two tools solve different problems. robots.txt controls whether a page gets crawled (visited by the bot). The noindex meta tag controls whether a visited page gets added to the search index. A page blocked in robots.txt will not be crawled, but it can still appear in search results if other sites link to it. For guaranteed removal from search results, use a noindex tag — not robots.txt alone.

Googlebot allocates a limited number of page crawls per site per day — your "crawl budget." For large sites, if Googlebot wastes this budget on duplicate URLs, paginated archives, or low-value parameters, important pages may go un-crawled. Using robots.txt to block low-value sections like /tag/, /search/, and URL parameters helps concentrate your crawl budget on the pages that matter most.

User-agent specifies which crawler the following rules apply to. Use User-agent: * to target all crawlers. Use a specific name like User-agent: Googlebot to target only Google's crawler, or User-agent: GPTBot to target OpenAI's training bot. Each User-agent block applies its rules until the next blank line. More specific blocks take precedence over the wildcard block.

Disallow: /path/ tells the crawler not to visit any URL starting with that path. Allow: /path/ explicitly permits a path even when a broader Disallow rule would block it. Key patterns: "Disallow: /" blocks your entire site (dangerous). "Disallow: /admin/" blocks the admin directory. "Disallow: " (empty) allows everything. "Disallow: /private/" + "Allow: /private/public.html" blocks all of /private/ except one page. Path matching is prefix-based: Disallow: /blog blocks /blog, /blog/, and /blog/post-1.

The Sitemap directive points crawlers to your XML sitemap. Example: Sitemap: https://example.com/sitemap.xml. This is not a restriction — it's a discovery hint that helps bots find your content faster. You can include multiple Sitemap lines for separate sitemaps. This generator automatically appends this directive when you enter a sitemap URL.

Crawl-delay: 10 tells supporting bots to wait 10 seconds between page requests, reducing server load from aggressive crawlers. Note: Googlebot ignores this directive entirely. To control Googlebot's crawl rate, use Google Search Console → Settings → Crawl Rate. Bingbot, Baiduspider, and most non-Google crawlers do honor Crawl-delay.

The single most damaging mistake: Disallow: / under User-agent: * blocks all crawlers from your entire site. This removes your website from Google entirely. Always double-check this rule before uploading. If your site suddenly disappears from search results, check robots.txt first.

Blocking /wp-content/ or other asset folders prevents Googlebot from loading your pages' styles and scripts. Google needs to render pages like a browser to evaluate their content and user experience. Blocked assets can result in lower rankings because Google cannot see what your pages look like to real users.

Robots.txt is publicly visible at yourdomain.com/robots.txt. Any content listed under Disallow: is broadcast to the world, including malicious bots that don't follow the protocol. Never use robots.txt to "hide" sensitive content — it actually advertises the path to anyone looking. Use proper authentication and server-level access controls for genuinely private content.

WordPress auto-generates a basic robots.txt if none exists. To use your own: upload the downloaded robots.txt file to the root directory of your WordPress installation via FTP or your hosting file manager. Alternatively, use an SEO plugin: in Yoast SEO go to Tools → File editor; in Rank Math go to General Settings → Edit robots.txt. Both let you paste your custom rules directly without FTP access.

Since 2021, Shopify stores support a customizable robots.txt.liquid template. Go to your Shopify Admin → Online Store → Themes → Actions → Edit code. Find robots.txt.liquid in the Templates folder. Replace the content with your custom rules or add sections as needed. Changes take effect immediately.

For static sites (Next.js, Astro, Hugo, plain HTML), place the robots.txt file in your public/ or static/ root directory so it is served at the domain root. For other CMS platforms like Wix or Squarespace, check your platform's SEO settings panel — most modern platforms include a dedicated robots.txt editor under their SEO or advanced settings.