Crawl and indexing errors are the reasons Google could not fetch a page or chose not to store it, and the Page indexing report in Google Search Console lists every one of them. A page that is not indexed cannot rank, so this report is where a technical SEO check starts.
What is the Page indexing report?
Open Indexing, then Pages in Google Search Console. The report splits the URLs Google knows about into Indexed and Not indexed, and lists a reason for each group of not-indexed URLs. A Source column tells you whether the cause sits on your site or with Google. Google’s documentation says to prioritise issues from the “Website” source, because the rest are largely outside your control.
Not every “Not indexed” row is a problem. Many are correct: a redirect, an alternate page or a page you blocked on purpose.
Which indexing statuses need a fix?
Statuses below come from Google’s Page indexing documentation.
| Status | What it means | What to do |
|---|---|---|
| Server error (5xx) | Your server returned a 500-level error | Check host health and the Crawl stats report, and speed up slow database-driven pages |
| Redirect error | A redirect chain too long, a loop or a bad URL in the chain | Fix the chain (see redirects) |
| URL blocked by robots.txt | robots.txt stops Googlebot | Remove the rule if the page should rank. To keep a page out of results, unblock it and use noindex (see robots.txt and noindex) |
| URL marked ‘noindex’ | A noindex tag or header is present | Remove it if the page should be indexed |
| Soft 404 | The page says “not found” or is nearly empty but returns status 200 | Return a real 404, or add real content |
| Not found (404) | The URL returned 404 | Fix only 404s you link to or list in a sitemap. Use a 301 redirect if the page moved (see custom 404 page) |
| Blocked due to unauthorized request (401) or access forbidden (403) | Googlebot sends no credentials | Allow Googlebot in, after verifying its identity, or accept that the page stays private |
| Crawled, currently not indexed | Google fetched the page and has not indexed it | Improve the page (see below) |
| Discovered, currently not indexed | Google knows the URL and has not crawled it yet | Check server health, link to the page, add it to the sitemap |
These statuses need no fix: Alternate page with proper canonical tag, Page with redirect, and Duplicate without user-selected canonical (Google says it is fine unless it picked the wrong canonical). For the canonical statuses, see canonical tags.
The report also has two warnings: Indexed, though blocked by robots.txt (remove the block and add noindex if you want the page out) and Page indexed without content (Google could not read the content, so inspect the page).
How do you fix crawl and indexing errors, step by step?
- Open the report and sort by count. Start with the reason that holds the most URLs, and ignore the harmless ones listed above.
- Click a reason and inspect examples. Select a URL and open URL Inspection. It shows the last crawl, the canonical Google chose and whether the page is blocked. Run Test live URL to see what Googlebot gets right now.
- Fix the cause at template level. A 5xx on a template, a bad redirect rule or a noindex setting in the CMS affects hundreds of URLs at once. Repair the rule, not each page.
- Fix “Crawled, currently not indexed” with content. Google’s documentation gives no switch for it. Confirm you want the page indexed, make its content clearly different from similar pages (or merge it into one page with a canonical), and link to it from pages that are already indexed.
- Fix “Discovered, currently not indexed” with server and links. Google says it postpones crawling when it expects to overload the site. Check availability in Crawl stats, keep URL parameters few and make sure the URL sits in your XML sitemap and has internal links.
- Click Validate fix. Fix all instances first, because validation stops if one remains. Google says validation takes up to about two weeks and emails you the result. Click only once and wait for it to finish.
- Request indexing for urgent pages. In URL Inspection, use Request indexing for a handful of important URLs. Do not bulk-request your whole site.
- Repeat monthly. New templates, plugins and migrations create new errors. Put a monthly look at this report in your calendar.
Does crawl budget matter for your site?
Crawl budget is the set of URLs Google can and wants to crawl, which depends on how much load your server handles and how worthwhile Google finds your URLs. Google’s crawl budget guide is written for sites with 1 million or more unique pages that change weekly, sites with 10,000 or more pages that change daily, and sites with many URLs stuck in “Discovered, currently not indexed”. Google calls these figures a rough guide, not exact thresholds.
A site below that size does not need crawl budget work. Keep the sitemap current and read this report. If you are in the range, Google recommends: consolidate duplicate content, block unimportant URLs with robots.txt, return 404 or 410 for pages removed for good, fix soft 404s, and speed up your server. Google also says not to use noindex to save crawl budget, because Googlebot must still request the page to see the tag.
How do you measure the result?
- Indexed count on the Page indexing report: it should rise toward the number of URLs in your sitemap.
- Error count by reason: each fixed template shows as a step down. Check the validation status (Started, Looking good, Passed or Failed).
- Sitemap filter: switch the report to “All submitted pages” to see only the URLs you put in your sitemap. Every one of them should be indexed.
- Crawl stats (Settings): watch the share of 5xx responses and average response time. Both should fall after server fixes.
What mistakes should you avoid?
- Panicking over every “Not indexed” row. Redirects and alternates are normal.
- Listing broken or redirecting URLs in your sitemap. The sitemap should hold only live, canonical, indexable pages.
- Using robots.txt to remove pages from Google. A blocked URL can stay in results, and the report then shows “Indexed, though blocked by robots.txt”.
- Validating before the fix is complete. One remaining instance fails the whole run.
- Resubmitting “Crawled, currently not indexed” pages again and again. Google says you need not resubmit them. Improve the page instead.
- Redirecting every 404 to the homepage. Google treats that as a soft 404. Use a 301 to the closest matching page, or let it return a clean 404.
Where can you learn more?
The free digital marketing module of the Fe/male Switch startup school covers setting up Search Console for a new startup site. Next steps on SEO100: fix broken links that cause 404 rows, then rerun the report after two weeks.