Where do I upload my robots.txt file?

The robots.txt file must be placed at the root of your domain: https://yourdomain.com/robots.txt. It cannot be in a subdirectory. For most CMS platforms (WordPress, Shopify, Webflow), there is a built-in robots.txt settings field — check your SEO plugin or platform settings.

Does robots.txt affect SEO rankings?

Directly, no — robots.txt only controls crawl access. But if you accidentally block important pages, Googlebot cannot read and rank them, which severely impacts your rankings. Correctly configured, it helps crawlers focus on your important pages and saves crawl budget.

Can I block specific bots like GPTBot or CCBot?

Yes. AI crawlers from OpenAI (GPTBot), Common Crawl (CCBot), and others follow robots.txt. Add a specific User-agent block for each bot you want to restrict. Note: some bots may not respect robots.txt, but major ones from Google, Bing, and OpenAI do.

What happens if I block everything with Disallow: /?

Your site will be crawled by no bots — including Googlebot. This effectively removes your site from search engines. It's the single most dangerous misconfiguration in technical SEO. Never use Disallow: / on a live production site you want indexed.

Technical SEO

Robots.txt Generator

Build a valid robots.txt file visually — no coding required. Control crawlers, protect private paths, and add your sitemap in seconds.

How to use this tool3 quick steps

Enter disallow paths
List any paths you want to block crawlers from — one per line. Common examples: /admin, /private, /api. Leave empty to allow everything.
Add your sitemap URL (optional)
Pointing crawlers to your sitemap speeds up discovery. Use the full absolute URL ending in sitemap.xml.
Copy and deploy
Save the output as robots.txt in the root of your domain. Every site should have one.

InputConfiguration· Paths and sitemap URL

Disallow paths(one per line)

Paths you want to block search engine crawlers from accessing. Leave empty to allow all paths.

Sitemap URL(optional)

Adding your sitemap URL helps search engines discover your pages faster.

OutputGenerated robots.txt· Copy and save as robots.txt

robots.txt

User-agent: *
Allow: /
Disallow: /admin

Copy this output and save it as robots.txt in the root of your website.

Use this with

See all 9 tools

Robots.txt Tester

Is this URL allowed?

Sitemap Generator

Build valid sitemap.xml

XML Sitemap Parser

Analyze sitemap structure & issues

robots.txt Guide

What is robots.txt and why it matters for SEO

The robots.txt file is a simple text file placed at the root of your website that tells search engine crawlers which pages they are allowed to visit. Misconfiguring it can block your entire site from Google — making it one of the highest-impact technical SEO files you'll ever edit.

How robots.txt works

When a crawler like Googlebot visits your site, it first requests /robots.txt. If the file exists, it reads the rules and obeys them (in most cases). If not, it crawls everything freely.

User-agent directive

The User-agent field specifies which crawler the rule applies to. Use * to apply to all crawlers, or specify "Googlebot", "Bingbot", "GPTBot", etc. for targeted rules.

Disallow vs. Allow

Disallow blocks a path from being crawled. Allow explicitly permits a path even within a blocked parent directory. Most sites only need Disallow entries.

Sitemap directive

The Sitemap directive in robots.txt tells crawlers where to find your XML sitemap. This helps Googlebot discover all your pages faster — especially important for large sites.

robots.txt ≠ noindex

Blocking a URL in robots.txt prevents crawling but does NOT remove it from the index. Pages can still appear in search results if other sites link to them. Use meta noindex on the page itself to de-index.

Common paths to block

/admin — admin dashboard. /api — server endpoints. /private — private documents. /wp-admin — WordPress admin. /checkout — e-commerce checkout pages (often thin content).

Pro Tips

Always test first

After uploading your robots.txt, test it in Google Search Console under "robots.txt Tester" to confirm your rules work as expected before crawlers visit.

Never block CSS or JS

Blocking your CSS and JavaScript files prevents Googlebot from rendering your pages correctly, which can hurt rankings. Only block HTML paths and content you genuinely want hidden.

Add your sitemap

Always include the Sitemap: directive pointing to your XML sitemap. This alone can speed up Google's discovery of your new pages by days.

Frequently Asked Questions

Where do I upload my robots.txt file?: The robots.txt file must be placed at the root of your domain: https://yourdomain.com/robots.txt. It cannot be in a subdirectory. For most CMS platforms (WordPress, Shopify, Webflow), there is a built-in robots.txt settings field — check your SEO plugin or platform settings.
Does robots.txt affect SEO rankings?: Directly, no — robots.txt only controls crawl access. But if you accidentally block important pages, Googlebot cannot read and rank them, which severely impacts your rankings. Correctly configured, it helps crawlers focus on your important pages and saves crawl budget.
Can I block specific bots like GPTBot or CCBot?: Yes. AI crawlers from OpenAI (GPTBot), Common Crawl (CCBot), and others follow robots.txt. Add a specific User-agent block for each bot you want to restrict. Note: some bots may not respect robots.txt, but major ones from Google, Bing, and OpenAI do.
What happens if I block everything with Disallow: /?: Your site will be crawled by no bots — including Googlebot. This effectively removes your site from search engines. It's the single most dangerous misconfiguration in technical SEO. Never use Disallow: / on a live production site you want indexed.