robots.txt generator
Build a valid robots.txt file. Pick a preset, add your rules, then copy or download it.
Runs 100% in your browserHow to create a robots.txt file
- Pick a starting point. Choose a preset (allow everything, block everything, or start blank) then adjust.
- Add your rules. List the paths to disallow or allow, and add your sitemap URL.
- Save as robots.txt. Copy or download the file and upload it to the root of your site.
How robots.txt works
robots.txt is a plain-text file at your domain root — it must live at exactly
/robots.txt, because crawlers fetch that fixed path before reading anything else. It
follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022: each group begins with one or
more User-agent lines, then Disallow and Allow rules. An empty
Disallow: permits everything; Disallow: / blocks the whole site. When rules
conflict, Google and Bing apply the most specific (longest) matching path rather than the
first one written, so an Allow for a sub-folder can carve an exception out of a broader
Disallow. Paths support * wildcards and the $ end-of-URL anchor.
What it can and can't do
The most common and costly misunderstanding: robots.txt controls crawling, not
indexing. Disallow stops well-behaved bots from fetching a URL, but if
other pages link to it that URL can still surface in results — shown without a snippet, because the
crawler was never allowed to read it. To actually keep a page out of the index you must allow
crawling and add a <meta name="robots" content="noindex"> tag (or an
X-Robots-Tag header). Equally, never disallow your CSS and JS: Google renders pages like a
browser, and blocking those resources can break how it sees your layout. Note too that the file is
public — don't list secret paths here, as you're handing curious visitors a map.
Presets, crawl-delay and testing
The presets above cover the usual cases: open the whole site, block everything (useful on a staging
server), or the common pattern of blocking admin and script folders while allowing the rest. The
optional Crawl-delay is honoured by Bing and others but ignored by Google — set
the crawl rate for Googlebot in Search Console instead. Adding a Sitemap: line (you can
include several) helps engines discover your URLs. Whatever you produce, run it through Search
Console's robots.txt tester before you rely on it: a single stray Disallow: / has
de-indexed entire sites.
Frequently asked questions
- robots.txt is a plain-text file at the root of your site (
example.com/robots.txt) that tells search-engine crawlers which paths they may or may not request. It controls crawling, not indexing — use anoindexmeta tag to keep a page out of results. - At the very root of your domain, reachable at
https://yourdomain.com/robots.txt. It only applies to the host and protocol it is served from. - No. Disallow stops well-behaved crawlers from fetching the page, but a disallowed URL can still appear in results if other pages link to it. To remove a page from the index, allow crawling and add
<meta name="robots" content="noindex">. - Yes — adding a
Sitemap:line helps search engines discover all your URLs. You can include more than one sitemap line.