Skip to content
LIVETOOLSLAB
All Tools

Robots.txt Generator

A robots.txt file tells compliant crawlers which parts of a site they may crawl. It lives at the root of your domain (example.com/robots.txt) and contains User-agent, Allow, Disallow and Sitemap lines.

Create a robots.txt file

🤖

Robots.txt Generator

Create a custom robots.txt file for search engine crawlers.

Enter one path per line. Example: /admin/
Enter one path per line. Example: /public/
Enter your complete sitemap URL, including https://
Your robots.txt
Important: robots.txt controls crawling instructions for compliant crawlers. It is not a security mechanism and should not be used to protect private or sensitive files. Google also notes that a disallowed URL can potentially still appear in search results even when its content cannot be crawled.

How to use it

  1. Start from Allow All, Block All or Custom Rules.
  2. Choose the user-agent: all crawlers (*) or a specific bot such as Googlebot, Googlebot-Image, Bingbot or DuckDuckBot.
  3. List paths to disallow and allow, one per line.
  4. Add your sitemap URL, select Generate, then copy or download robots.txt.

Example for a WordPress site

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/

Sitemap: https://example.com/wp-sitemap.xml

Important limits

  • robots.txt is not security. Anyone can read it, and badly behaved bots ignore it. Protect private content with a login.
  • Blocking a URL doesn’t guarantee it stays out of search results. Google can index a blocked URL it finds through links. Use a noindex tag on a crawlable page to keep it out.
  • One mistaken Disallow: / can stop a whole site from being crawled. Double-check before uploading.

How matching works

  • Rules match the start of the URL path: Disallow: /private also blocks /private-notes/.
  • Add a trailing slash to block only a folder: Disallow: /private/.
  • Google supports * as a wildcard and $ to mark the end of a URL, e.g. Disallow: /*.pdf$.
  • When Allow and Disallow both match, Google applies the most specific (longest) rule.
  • A crawler follows only the group for its most specific user-agent, so a Googlebot group overrides the * group for Googlebot.

Testing before you upload

After uploading, open https://yourdomain.com/robots.txt to confirm it’s live. Google Search Console’s robots.txt report shows the version Google fetched and any parsing errors. Check one important URL with the URL Inspection tool to make sure it isn’t blocked by accident.

Frequently asked questions

To the root of your domain so it’s reachable at https://yourdomain.com/robots.txt. WordPress also generates a virtual one, and SEO plugins let you edit it.

Commonly yes, while allowing /wp-admin/admin-ajax.php, which some front-end features use.

Related tools

More in SEO Tools · All tools