robots.txt is a file that tells search engine crawlers which pages or sections of your site should not be crawled or indexed.

Does robots.txt block pages from Google?

Robots.txt only prevents crawling, not indexing. Pages can still appear in search results if linked from other sites. Use noindex meta tags to prevent indexing.

What is the User-agent directive?

User-agent specifies which crawler the rules apply to. Use * for all crawlers, or specific names like Googlebot or Bingbot for targeted rules.

Should I block CSS and JavaScript files?

No, Google recommends allowing access to CSS and JS files so it can properly render and understand your pages for better indexing.

SEO Tools

Robots.txt Generator

Generate a robots.txt file for your website. Control search engine crawling with an easy-to-use generator.

Quick Presets

Rule 1

User-agent

Disallow Paths

Allow Paths (override disallow)

/

Sitemap URLs

Advanced Options

Crawl Delay (seconds):Delay between requests (not supported by all crawlers)

Generated robots.txt

User-agent: *
Allow: /

Upload this file to the root directory of your website (e.g., https://example.com/robots.txt)

Looking for more?

Automate your workflow with Browzey - no coding required.

Related Tools

SEO Tools

Canonical Tag Checker

Check canonical tags on any webpage. Identify SEO issues with self-referencing and cross-domain canonicals.

SEO Tools

Meta Tag Analyzer

Analyze meta tags on any webpage. Check title, description, Open Graph, and Twitter cards for SEO issues.

Data Extraction

Sitemap URL Extractor

Extract all URLs from XML sitemaps. Parse sitemap.xml files and get a complete list of pages for SEO analysis. Free online tool.

About This Tool

Frequently Asked Questions

Related Tools

Canonical Tag Checker

Meta Tag Analyzer

Sitemap URL Extractor