What robots.txt Can and Can't Do (And Why It Won't Hide a Page)

I have watched more than one team block a staging directory in robots.txt, high-five, and then find the URL sitting in Google's index two weeks later. Nothing went wrong with the file. The file did exactly what it says on the tin. The problem is that almost everyone reads Disallow as "hide this," when what it actually means is "don't fetch this." Those are wildly different instructions, and once that clicks, the whole protocol stops feeling arbitrary.

Crawling and indexing are two separate jobs

A search engine does two things with your site, in order. First it crawls: it fetches URLs and reads what comes back. Then it indexes: it decides which of those URLs deserve to be in the searchable database, and what they are about.

robots.txt only speaks to the first job. It is a fetching policy. It has no vocabulary at all for the second one.

Disallow (robots.txt)   ->  "Do not fetch this URL."
noindex (meta or header) -> "Fetch it, read it, then keep it
                             out of the index."

Which leads to the trap. If you block a URL and something out there links to it, a search engine can learn the URL exists without ever fetching it. It has a URL, it has anchor text from the linking page, and it has no content because you told it not to look. So it can list the bare URL in results with no description. You did not hide the page. You made it show up badly.

The fix is counterintuitive: to keep a page out of the index, you have to let crawlers fetch it so they can see the noindex instruction.

<!-- in the HTML head -->
<meta name="robots" content="noindex">

# or as a response header, which also works for PDFs,
# images, and anything else that has no HTML head
X-Robots-Tag: noindex

Block that same URL in robots.txt and the noindex is never read. The two directives cancel each other out. Pick one.

The directives Google actually honors

robots.txt was a handshake agreement for about 28 years before it became a real standard. It was written up as RFC 9309 in September 2022, which finally pinned down the parts everyone had been implementing slightly differently.

What the major crawlers do with each directive

Directive	Google	Notes
User-agent	Honored	Picks the single most specific matching group. Groups are not merged.
Disallow	Honored	Blocks fetching. Says nothing about indexing.
Allow	Honored	Carves exceptions out of a broader Disallow.
Sitemap	Honored	Independent of user-agent groups. Needs an absolute URL.
Crawl-delay	Ignored	Bing honors it. For Google, set the crawl rate in Search Console instead.
noindex	Ignored	Google dropped support on September 1, 2019. It was never in any standard.

That noindex row catches people out because you still find it in blog posts from 2015. It has not worked for years. If you have it in a file, it is doing nothing.

Matching rules, and why order does not save you

Two wildcards are supported, and both are in RFC 9309:

• * matches any sequence of characters.
• $ anchors the match to the end of the URL.

Here is the part that surprises people: Google does not evaluate rules top to bottom. It picks the rule with the longest matching path, and if a Disallow and an Allow tie on length, Allow wins. Rewriting your file in a different order changes nothing.

User-agent: *
Disallow: /downloads/
Allow: /downloads/public/

# /downloads/report.pdf        -> blocked  (only /downloads/ matches)
# /downloads/public/logo.png   -> allowed  (longer match wins)

Two more details that bite:

• Paths are case-sensitive. Disallow: /Admin/ does not touch /admin/.
• Scope is one origin. A robots.txt is tied to a scheme, host, and port. The file at https://example.com/robots.txt does not govern http://example.com, and it does not govern shop.example.com. Subdomains need their own file.

What happens when robots.txt itself breaks

This is the failure mode nobody plans for, and the behavior is not symmetrical:

200 + rules   ->  rules apply
404 / 410     ->  no restrictions; crawl everything
5xx           ->  Google temporarily treats the whole site
                  as disallowed

Read that last line again. A flaky robots.txt endpoint does not fail open, it fails closed. If your server returns 500 on that path during a deploy, Googlebot backs off the entire site while the error lasts. After a long stretch of failures Google falls back to its last cached copy, and if there is no usable cached copy it will go back to assuming no restrictions. Either way, serving a real 404 for a site with nothing to hide is strictly safer than serving an unreliable 500.

One more limit worth knowing: Google reads the first 500 kibibytes of the file and ignores the rest. If you are anywhere near that, you are using robots.txt for a job that belongs to noindex or to your routing layer.

A file that covers the common cases

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sessionid=
Allow: /

Sitemap: https://example.com/sitemap.xml

Short, boring, and correct. That is the goal. Note what is not in there: no attempt to block a private area (that needs authentication, not a polite request), and no attempt to deindex anything.

Hand-writing these is fine until you need a per-agent group and a couple of wildcards, at which point a typo silently blocks the wrong thing. Our robots.txt generator builds the file from readable rules and runs entirely in your browser, and the sitemap generator produces the file that Sitemap: line should point at.

The one-paragraph version

robots.txt controls fetching, not visibility. Use Disallow to keep crawlers away from things that waste their time: faceted search, session URLs, internal endpoints. Use noindex to keep a page out of results, and leave that page crawlable so the instruction gets read. Use real authentication for anything that genuinely must stay private, because robots.txt is a public file that politely asks well-behaved bots to cooperate, and the badly-behaved ones read it for a list of interesting places to look.