Build a correct robots.txtwithout nuking your rankings.
One stray slash in this file can hide your entire site from Google. Pick a preset or paste your paths, get a clean, valid robots.txt, and a plain-English warning before you ship the one mistake that quietly deindexes you.
Your crawl rules
Pick a preset, or choose Custom to list your own paths. Empty Disallow means allow everything; a lone Disallow: / blocks the whole site.
Your robots.txt
User-agent: * Disallow: Sitemap: https://yoursite.com/sitemap.xml
Pre-flight checks
Ready to publish
Save this as /robots.txt at your domain root (yoursite.com/robots.txt), served as plain text, lowercase filename.
Your move
A clean robots.txt is table stakes. Rankings are the real work.
Your robots.txt is valid and points crawlers at your sitemap.
Technical SEO keeps you out of the ditch; it doesn't win the race. If your site is crawlable but still not pulling traffic, the problem is upstream, in the offer, the pages, or the content strategy. Bring me a site that's indexed but invisible and I'll show you, free, where the actual leaks are.
Plain English
What robots.txt does, and the three things it doesn't.
robots.txt is a plain text file that lives at the root of your domain (yoursite.com/robots.txt) and tells well-behaved crawlers, Googlebot, Bingbot, and friends, which parts of your site they're welcome to crawl. It's the first file most bots request before they touch anything else, which is exactly why a single bad line in it carries so much weight.
Here's what trips people up: robots.txt controls crawling, not indexing. A page you Disallow can still show up in search results (as a bare URL with no description) if other sites link to it, because Google never crawled it to read your noindex tag. It is also not a security tool, anyone can read your robots.txt, so listing /secret-admin/ in it is a map to exactly where you don't want people looking. And it doesn't remove anything already indexed; for that you need noindex or the URL removal tool.
So treat robots.txt as a crawl-budget and politeness file, not a lock. Use it to keep bots out of low-value paths (faceted search, internal scripts, staging junk) and to point them at your sitemap. Use meta robots noindex (or an X-Robots-Tag header) when you actually want a page kept out of the index. This tool builds the file; it'll also flag the classic footgun, a lone Disallow: / that blocks the whole site, before you copy it.
The formula
User-agent: * → Disallow / Allow rules → Crawl-delay (optional) → Sitemap: URL
A WordPress-safe file: 'User-agent: *' then 'Disallow: /wp-admin/' (keep bots out of the dashboard) and 'Allow: /wp-admin/admin-ajax.php' (but let the one endpoint themes need through), a blank line, then 'Sitemap: https://yoursite.com/sitemap.xml'. Empty value, 'Disallow:', means allow everything; 'Disallow: /' means block everything.
Your robots.txt looks off. Here's the fix.
It contains a bare 'Disallow: /'.
That single line blocks crawlers from your entire site. Unless you're intentionally hiding a staging environment, delete it now, this is the one mistake that tanks rankings overnight.
Traffic dropped after a launch.
Check the live file first. Dev sites ship with 'Disallow: /' to stay out of the index, and it gets pushed to production constantly. Confirm production reads 'Disallow:' (empty), not 'Disallow: /'.
A blocked page still shows in Google.
Expected. robots.txt stops crawling, not indexing. To remove it, allow the crawl and add a meta robots noindex tag so Google can actually read the instruction, then request removal.
You listed admin or private paths.
robots.txt is public. You just published a directory of where your sensitive URLs live. Protect those with authentication, not a Disallow line, and avoid naming them here.
How to write robots.txt that helps instead of hurts
01Place it at the exact root
It must live at yoursite.com/robots.txt, lowercase, nothing else works. A file in a subfolder or a subdomain's root won't cover the main domain.
02Default to open
Most sites want everything crawlable. If you're unsure, an empty 'Disallow:' (allow all) plus a sitemap line is the safe baseline. Block surgically, not broadly.
03Block low-value, not whole sections
Faceted-search URLs, internal cart/checkout, and duplicate parameter pages waste crawl budget. Disallow those specific paths, never the parent everything lives under.
04Always declare your sitemap
Adding 'Sitemap: https://yoursite.com/sitemap.xml' helps every crawler find every page. It's the cheapest indexing win in the file and most people forget it.
05Use noindex for real removal
Crawling and indexing are different jobs. If a page must stay out of results, let bots crawl it and serve a meta robots noindex, robots.txt alone can't do that.
06Mind the trailing slash
'Disallow: /blog' blocks /blog and /blog-post; 'Disallow: /blog/' blocks only the folder. The slash changes the meaning, be deliberate.
07Test before you trust
Paste your file into Google Search Console's robots.txt tester and check a few real URLs. A typo'd directive fails silent, bots just ignore the broken line.
08Skip crawl-delay for Google
Googlebot ignores Crawl-delay entirely (control it in Search Console). It's only useful for Bing and smaller crawlers hammering a fragile server.
The vocabulary
- User-agent
- The crawler a block of rules applies to. '*' means all bots; you can target specific ones like 'Googlebot'.
- Disallow
- Tells the listed user-agent not to crawl paths starting with the given value. Empty ('Disallow:') means allow everything.
- Allow
- Explicitly permits a path, used to carve an exception out of a broader Disallow rule (e.g. allow one file inside a blocked folder).
- Crawl-delay
- Seconds a bot should wait between requests. Honored by Bing and others; ignored by Googlebot.
- Sitemap
- An absolute URL pointing crawlers to your XML sitemap. Lives at file level, not inside a user-agent block.
- Crawl budget
- The number of pages a bot will crawl on your site in a given window. robots.txt helps you spend it on pages that matter.
Robots.txt questions, straight answers
At the root of your domain, exactly: https://yoursite.com/robots.txt, lowercase filename, served as plain text. It can't live in a subfolder, and a robots.txt on a subdomain only governs that subdomain. Save the output of this tool as a file named robots.txt and upload it to your web root.
Keep going
A calculator tells you what. A call tells you what to do about it.
Send me the account behind these numbers. I'll tell you straight where the money's leaking and what I'd fix first — free, and you keep it whether you hire me or not.