Skip to content
All Tools

Robots.txt Generator

A robots.txt generator helps you create a robots.txt file for your website without writing crawler rules manually. Use this free robots.txt file generator to create and customize instructions for search engine crawlers, such as Googlebot. You can control which website paths crawlers are allowed or disallowed from accessing, generate a custom robots.txt file, and download it for upload to your website’s root directory. This robots.txt creator makes it easier to prepare crawler instructions for websites, including WordPress sites.

Create a robots.txt file

🤖

Robots.txt Generator

Create a custom robots.txt file for search engine crawlers.

Enter one path per line. Example: /admin/
Enter one path per line. Example: /public/
Enter your complete sitemap URL, including https://
Your robots.txt
Important: robots.txt controls crawling instructions for compliant crawlers. It is not a security mechanism and should not be used to protect private or sensitive files. Google also notes that a disallowed URL can potentially still appear in search results even when its content cannot be crawled.

How to Use the Robots.txt Generator

Follow these steps to create a robots.txt file using this free robots.txt generator:

  1. Choose a starting configuration: Select Allow All, Block All, or Custom Rules, depending on how you want search engine crawlers to access your website.

  2. Select a user-agent: Choose all crawlers (*) or a specific search engine bot, such as Googlebot, Googlebot-Image, Bingbot, or DuckDuckBot.

  3. Add crawler rules: Enter the website paths you want crawlers to disallow or allow. Add each path on a separate line.

  4. Add your sitemap URL: Enter your XML sitemap URL to help search engines discover your website’s preferred URLs.

  5. Generate your file: Click Generate to create your custom robots.txt file.

  6. Copy or download: Copy the generated robots.txt code or download the file, then upload it to your website’s root directory.

This robots.txt creator makes it easier to generate robots.txt files for different crawler configurations without manually writing every rule. Review the generated instructions before publishing them to avoid accidentally blocking important website sections.

Example Robots.txt File for a WordPress Site

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://example.com/wp-sitemap.xml

Understanding these robots.txt rules

  • User-agent: * – Applies the following crawler instructions to all compliant search engine bots.

  • Allow: /wp-admin/admin-ajax.php – Allows crawlers to access the WordPress AJAX endpoint.

  • Disallow: /wp-admin/ – Requests crawlers not to access the WordPress administration directory.

  • Sitemap: – Provides the location of your XML sitemap to help search engines discover your website URLs.

Important WordPress configuration note

Replace https://example.com/wp-sitemap.xml with your actual sitemap URL. WordPress provides a built-in XML sitemap, but SEO plugins such as Yoast SEO may provide their own sitemap index URL.

Before uploading a custom robots.txt file, check your existing WordPress configuration to avoid conflicting crawler rules. The robots.txt file should be accessible at your domain’s root URL, such as https://example.com/robots.txt.

Important Robots.txt Limitations

A robots.txt file helps manage how compliant search engine crawlers access your website, but it does not provide complete control over indexing, security, or website access. Understand these limitations before using a robots.txt generator.

  • Robots.txt is not a security tool: Anyone can view your robots.txt file, and malicious bots may ignore its instructions. Protect private pages, files, and sensitive information with proper authentication and access controls instead of relying on Disallow rules.

  • Blocking a URL does not guarantee deindexing: Google may still show a blocked URL in search results if it discovers that URL through external or internal links. If you want a page excluded from search results, allow Googlebot to crawl it and use an appropriate noindex directive.

  • Avoid accidentally blocking your entire website: A rule such as Disallow: / tells compliant crawlers not to crawl any URL on the website for the specified user-agent. Check your generated robots.txt rules carefully before uploading the file.

  • Different search engines may interpret rules differently: Although major search engines support common robots.txt directives, crawler behavior and supported features can vary. Review the relevant search engine documentation when using advanced rules.

  • Robots.txt does not remove existing indexed pages: Changing crawler rules does not automatically remove URLs that are already indexed. Indexing and removal require separate actions depending on your situation.

Before publishing a custom robots.txt file, test its rules and verify that important pages, resources, and sitemap locations are not unintentionally blocked.

How Robots.txt Rule Matching Works

Understanding how robots.txt rules match URL paths helps you create accurate crawler instructions and avoid blocking pages unintentionally.

  • Rules match the beginning of a URL path: A rule such as Disallow: /private can block /private, /private-notes/, and other paths beginning with /private.

  • Use a trailing slash for specific folders: If you want to block a folder named private, use Disallow: /private/. This avoids matching paths such as /private-notes/.

  • Use wildcards for flexible matching: Google supports * as a wildcard for matching any sequence of characters and $ to indicate the end of a URL path. For example, Disallow: /*.pdf$ blocks URLs ending in .pdf for the specified crawler.

  • The most specific matching rule takes priority: When Google encounters matching Allow and Disallow rules, it generally follows the rule with the longest matching path. If the matching rules have equal length, Google favors Allow.

  • Specific user-agent groups take priority: A crawler follows the group that most specifically matches its user-agent. For example, a Googlebot-specific group takes precedence over a general User-agent: * group for Googlebot.

Example of robots.txt rule matching

User-agent: *
Disallow: /private/

User-agent: Googlebot
Allow: /private/public-page/

In this example, the general rule blocks the /private/ directory for compliant crawlers, while the Googlebot-specific group provides its own instructions for Googlebot.

Review your robots.txt rules carefully before publishing them, especially when combining multiple user-agent groups, wildcards, and Allow or Disallow directives.

How to Test Your Robots.txt File Before and After Uploading

After using a robots.txt generator and uploading the file to your website, test it to make sure the crawler instructions are working as expected. A small mistake in your robots.txt configuration can prevent search engines from crawling important website pages.

Follow these checks:

  1. Open your robots.txt URL: Visit https://yourdomain.com/robots.txt in your browser to confirm that the file is publicly accessible and displays the latest rules.

  2. Check for formatting errors: Review the file for missing line breaks, incorrect user-agent names, misplaced directives, and unintended Allow or Disallow rules.

  3. Review Google Search Console: Check the available robots.txt reporting information in Search Console to identify fetching or parsing problems with the file Google has retrieved.

  4. Inspect important URLs: Use the URL Inspection tool in Google Search Console to check whether important pages can be crawled by Googlebot.

  5. Verify your sitemap location: Make sure the sitemap URL included in your robots.txt file is correct and accessible.

  6. Recheck after changes: Whenever you update your robots.txt file, open it again in your browser and review the latest crawler instructions.

Testing your custom robots.txt file helps identify configuration mistakes before they affect your website’s search engine crawling.

Frequently asked questions

Upload your robots.txt file to the root directory of your domain so search engines can access it at an address such as https://yourdomain.com/robots.txt. On WordPress websites, a virtual robots.txt file may be generated automatically, and SEO plugins or hosting tools may provide options to manage its rules.

Yes, it is common practice to disallow crawling of the WordPress administration directory using Disallow: /wp-admin/. However, you should generally allow /wp-admin/admin-ajax.php, as some WordPress themes and plugins use it for front-end functionality.

Use a robots.txt generator to select crawler settings, add Allow and Disallow rules, and include your sitemap URL. Generate the file, review the instructions, and download or copy the resulting code for your website.

A basic robots.txt file usually contains a user-agent declaration, crawler access rules, and optionally a sitemap URL. For example:

User-agent: *
Disallow: /private/

Sitemap: https://example.com/sitemap_index.xml

Replace the example sitemap address and directory rules with the correct settings for your website.

No. A website does not need a robots.txt file to appear in Google Search. However, creating one can help communicate crawler instructions and manage access to specific URL paths.

Not reliably. Robots.txt controls crawling, not guaranteed indexing. Google may still index a blocked URL if it discovers it through other links. Use a noindex directive on a crawlable page when you want to prevent it from appearing in search results.

Yes. You can create custom crawler rules for WordPress using a robots.txt generator and manage them through supported hosting tools or SEO plugins. Before making changes, check whether WordPress or an existing plugin already provides a virtual robots.txt file.

Yes. You can create specific rules for Googlebot using a user-agent declaration such as User-agent: Googlebot. These rules can specify which URL paths Googlebot is allowed or disallowed from crawling.

You can use a general User-agent: * group for common crawler instructions. However, you can also create separate user-agent groups for supported crawlers when different access rules are required.

Yes, this free robots.txt generator lets you create crawler instructions, customize Allow and Disallow rules, and generate a robots.txt file without manually writing the code.