A text file at your website's root that tells search engine crawlers which URLs they can and can't access. It's a standard part of web SEO.

Will robots.txt block pages from Google?

It prevents crawling, not indexing. To fully block a page, also use a noindex meta tag.

Where do I put the file?

At the root of your domain: https://example.com/robots.txt

Generate robots.txt files for your website. Control which pages search engines can crawl. Free online tool.

User-agent

Sitemap URL

Crawl-delay (seconds)

Output

User-agent: *
Disallow:

Generate a robots.txt file for your website. Add rules for different crawlers, specify disallowed paths, and include your sitemap URL.

Multiple user-agents

Add rules for Googlebot, Bingbot, or all crawlers.

Allow/disallow paths

Specify which directories to allow or block.

Sitemap link

Include your sitemap URL in the file.