Robots.txt Explained for Bloggers
Robots.txt controls what crawlers may fetch, not what appears in search. Mixing up the two is how blogs vanish from Google or keep pages they wanted gone.
Robots.txt is not a tool to remove pages from Google's search results. A robots.txt is a crawl map, not an indexing switch. The plain text file lives at the root of a host, such as https://example.com/robots.txt, and its rules apply only to that host, protocol and port. Google says it is used mainly to avoid overloading a site with requests.
What the file can contain
User-agent: *
Disallow: /search
Allow: /
Sitemap: https://example.com/sitemap.xmlGoogle supports four fields: user-agent (which crawler the group applies to), disallow and allow (paths it may not or may fetch) and sitemap (a fully qualified sitemap URL). Paths are case-sensitive. When rules conflict, Google applies the most specific one, meaning the longest path, and on a tie the least restrictive. Fields such as crawl-delay are ignored by Google, and the file size limit is 500 KiB.
Blocking is not noindex
A URL blocked in robots.txt can still appear in search results if other pages link to it, just without a description. To keep a page out of Google, use <meta name="robots" content="noindex"> in the page or an X-Robots-Tag: noindex HTTP header, or put the page behind a password. The noindex documentation adds a key condition: the page must not be blocked by robots.txt, because a crawler that cannot fetch the page never sees the noindex rule. Blocking and noindexing the same URL is therefore the most common self-defeating setup.
Platform defaults, testing and mistakes
On Blogger, the default file allows Google's AdSense crawler everything, disallows /search (label and search result pages), allows the rest and lists the blog's sitemap. Blogger Settings has a Crawlers and indexing section where Enable custom robots.txt replaces that default with custom content; the separate Visible to search engines switch sits under Privacy.
On WordPress with no physical robots.txt file, WordPress serves a virtual one: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, plus a line pointing to the core sitemap. The option under Settings → Reading, "Discourage search engines from indexing this site," now adds a noindex robots meta tag rather than a disallow rule. A real robots.txt file uploaded to the site root takes the place of the virtual one.
The old robots.txt Tester has been retired. Its replacement is the robots.txt report in Search Console, under Settings, which shows the files Google found for the top 20 hosts, when each was last crawled, any errors, and lets owners request a recrawl after an urgent fix. To check a single URL, run a live test in URL Inspection. Menu names in Blogger, WordPress and Search Console can change.
Mistakes that block too much:
- A
Disallow: /left over from a staging copy, which blocks the whole site. - Disallowing a page and adding
noindexto it at the same time. - Path typos or wrong letter case, which block the wrong URLs or none at all.
- Blocking CSS or JavaScript that Google needs to render and understand the page.
- A robots.txt URL that returns server errors: Google treats 5xx responses as a reason to hold back crawling, while a 4xx response is treated as if no robots.txt exists.
