robots.txt is a plain text file at the root of your site that tells crawlers which URLs they may fetch, and noindex is a directive that tells search engines not to show a page in results. The two get mixed up constantly, and the mix-up costs sites real traffic: Google’s documentation says robots.txt is for managing crawl traffic and is not a way to keep a page out of Google.
What is the difference between robots.txt and noindex?
robots.txt Disallow |
noindex |
|
|---|---|---|
| Controls | Crawling: whether the bot fetches the URL | Indexing: whether the page appears in results |
| Where it lives | One file at /robots.txt |
A meta tag in the page or an X-Robots-Tag HTTP header |
| Hides a page from Google? | No. A blocked URL can still appear, without a description, if other sites link to it | Yes, once Google recrawls the page and reads the tag |
| Works if the page is blocked in robots.txt? | n/a | No. The crawler never sees the tag |
Google also does not support noindex as a line inside robots.txt. Put it on the page.
Why do robots.txt and noindex matter for SEO?
Both protect the pages that earn traffic. A good robots.txt keeps Googlebot away from endless filter URLs, internal search results and admin paths, so crawling goes to the pages you care about. A correct noindex keeps thin pages (thank-you pages, tag archives, staging copies) out of results without a fight.
A wrong one does damage fast. One stray Disallow: / after a site relaunch blocks every page, and rankings drop as Google can no longer crawl the site.
What does a correct robots.txt look like?
User-agent: *
Disallow: /admin/
Disallow: /search
Disallow: /*?sessionid=
Sitemap: https://example.com/sitemap.xmlHow Google reads the file, per its documentation:
- Location: the top-level directory of the host. A file in a subfolder is never checked, and each subdomain needs its own file.
- Size: Google reads the first 500 KiB and ignores the rest.
- Supported fields:
user-agent,allow,disallowandsitemap. Other fields, such ascrawl-delay, have no effect on Google. - Matching: paths are case-sensitive,
*matches any sequence and$marks the end of a URL. When rules conflict, the most specific (longest) path wins, and the less restrictive rule applies on a tie. - Groups: a crawler follows the most specific
user-agentgroup that matches it, and the*group is not merged with it. - Errors: a 4xx status (except 429) counts as “no robots.txt”, so nothing is restricted. A 5xx status pauses crawling for 12 hours while Google retries, then falls back to the last good copy for up to 30 days.
- Caching: Google generally caches the file for up to 24 hours, so a fix takes up to a day to apply.
The format is standardised as RFC 9309 (Robots Exclusion Protocol). Compliance is voluntary: Google and other reputable crawlers follow it, and abusive bots ignore it, so it is not a security tool. For AI crawlers, see AI crawlers.
How do you set up robots.txt and noindex, step by step?
- Decide what each page needs. Pages that should rank: allow crawling, no noindex. Pages that should stay out of results: allow crawling and add noindex. Pages that waste crawl effort (filters, internal search, endless calendars): block with robots.txt.
- Write robots.txt. Keep it short. Block only paths that hold no value in search, and add your sitemap line (see XML sitemaps).
- Do not block CSS or JavaScript files. Google’s documentation warns that blocking resources makes a page harder for Google to understand if it needs them to render.
- Add noindex where needed. In the page head:
<meta name="robots" content="noindex">. For Google only, usename="googlebot". For PDFs and images, send the HTTP headerX-Robots-Tag: noindex. WordPress SEO plugins and most CMSs have a checkbox for this. - Keep noindex pages crawlable. If robots.txt blocks the URL, Google never sees the tag, and the page can stay in results.
- Check the live file. Open
https://yoursite.com/robots.txtin a browser. It must return status 200 and plain text. - Test in Search Console. The robots.txt report (Settings) shows the files Google found, when it last fetched them and any parse errors. The URL Inspection tool says whether a URL is blocked by robots.txt or carries noindex.
- Ask for a recrawl. Google applies noindex only after it recrawls the page, which its documentation says can take months for less important pages. Request indexing in URL Inspection to speed this up, and use the removals tool in Search Console for urgent cases.
How do you measure the result?
- The page indexing report lists “Blocked by robots.txt” and “Excluded by ‘noindex’ tag” with example URLs. Check that every URL in those two groups is there on purpose.
- Compare the count of indexed pages against the pages you want indexed. A drop right after a release points to a new rule or tag.
- Crawl the site with Screaming Frog before and after changes, and filter by “blocked” and “noindex”.
- Read the crawl stats in Google Search Console (Settings, Crawl stats) after blocking a large section: requests to that section should fall.
What mistakes should you avoid?
- Using Disallow to hide a page from Google. The URL can still be indexed from external links. Use noindex or a password.
- Adding noindex to a page robots.txt blocks. Google cannot see the tag.
- Shipping the staging file to production. A leftover
Disallow: /or site-wide noindex from development is a classic launch accident. Check both on launch day. - Blocking a page and canonicalising it. Google cannot read a canonical tag on a page it is blocked from fetching (see canonical tags).
- Pruning with noindex before you check traffic. Run content pruning decisions on data, then apply the tag.
- Treating robots.txt as protection for private files. Anyone can read the file and see the paths you list. Put real secrets behind a login.
Where can you learn more?
The free digital marketing module of the Fe/male Switch startup school shows how a founder checks that a new site is crawlable and indexed. Next steps on SEO100: add your sitemap to robots.txt, then work through the statuses in the page indexing report.