Why is your website having indexation problems?

Google doesn’t index a page when it can’t find it, isn’t allowed to crawl or index it, or decides it isn’t worth indexing. The fastest way to find out which one applies is Google Search Console: the URL Inspection tool shows the status of a single page, and the Page indexing report lists every reason Google gives for pages that aren’t indexed.

How do you check whether a page is indexed?

1. URL Inspection in Search Console. Paste the URL: you’ll see whether it’s on Google, which canonical Google chose, when it was last crawled, and whether crawling and indexing are allowed. Use “Test live URL” to check the current version. See Google’s guide.
2. Page indexing report. Shows how many pages are indexed and groups the others by reason. See the report.
3. site: operator (site:example.com) as a rough check only: results are approximate and can miss indexed pages.

For a new site or page: verify the domain in Search Console, submit an XML sitemap and use “Request indexing” in URL Inspection for key URLs (the number of requests is limited). For Bing and other engines, use Bing Webmaster Tools and IndexNow.

What do the Search Console statuses mean, and how do you fix them?

Status in Page indexing	What it means	What to do
Excluded by ‘noindex’ tag	The page or its X-Robots-Tag header says noindex	Remove noindex if the page should be indexed (check CMS settings, e.g. WordPress “Discourage search engines”)
Blocked by robots.txt	Google can’t crawl the URL	Remove the Disallow rule if the page should be indexed
Indexed, though blocked by robots.txt	URL is indexed (from links) without content	To remove it: allow crawling and add noindex
Page with redirect	URL redirects elsewhere	Fine for old URLs; link and list the final URL in sitemaps
Alternate page with proper canonical tag	Google indexes the canonical version	No action if intended
Duplicate, Google chose different canonical than user	Google disagrees with your canonical	Make the preferred URL clearly the main one: canonical, internal links, sitemap, redirects
Duplicate without user-selected canonical	Similar pages without a canonical	Add canonicals or merge pages
Not found (404) / Soft 404	Page doesn’t exist, or looks empty while returning 200	Return 404/410 for removed pages, 301 to replacements; add real content to soft 404s
Server error (5xx)	The server failed when Google crawled	Fix server stability; persistent 5xx errors lead to removal from the index
Discovered – currently not indexed	Google knows the URL but hasn’t crawled it yet	Improve internal links, sitemap and server speed; reduce low-value URLs (filters, parameters)
Crawled – currently not indexed	Google crawled the page and chose not to index it (for now)	Improve uniqueness and usefulness, merge thin pages, add internal links

What are the most common causes of indexing problems?

1. The page is blocked from indexing

A noindex directive in the meta robots tag or the X-Robots-Tag HTTP header (often set in .htaccess or server configuration). WordPress has a “Discourage search engines from indexing this site” option that’s easy to leave on after launch.

2. robots.txt is used instead of noindex

A Disallow rule in robots.txt stops crawling, not indexing. A blocked URL can still be indexed from links (without content), and Google can’t see a noindex on a page it isn’t allowed to crawl.

3. Several versions of the site work without redirects

If http/https or www/non-www versions all return 200, Google has to choose one, and signals are split. Redirect (301) all versions to one preferred address. Prefer the version that’s already indexed. See our HTTP traffic case study.

4. Pages are hard to find

Pages with no internal links (orphan pages) and missing from the sitemap may never be discovered. Every important URL should be linked from an indexed page and listed in the XML sitemap.

5. Duplicate or thin content

Google picks one canonical URL among duplicates (filters, parameters, print versions, near-identical product variants) and may not index pages with little unique content. Merge or improve them; see content audit.

6. Server errors and slow responses

If the server is slow or returns 5xx errors, Google slows down crawling, and persistent errors lead to URLs being dropped. See our server crash case study. Improve server response time (TTFB) rather than chasing a PageSpeed score.

7. Content only rendered by JavaScript

Google renders JavaScript, but rendering can be delayed and errors can hide content or links. Most AI crawlers don’t render JavaScript at all. Make sure key content and links are in the server-rendered HTML; check it in Chrome DevTools.

8. Wrong redirects

For permanent URL changes, use 301 redirects, not 302. Avoid redirect chains and redirecting removed pages to the homepage (often treated as soft 404).

9. Manual actions or security issues

Violations of Google’s spam policies (cloaking, hidden text, spammy links, scraped content) can lead to a manual action. A hacked or infected site appears in the Security issues report. Fix the problem and request a review in Search Console.

10. The domain has a bad history

A domain previously used for spam may need extra time and a clean, useful site to earn trust. Check its history (archive.org, backlinks) before buying.

Is it your site or a Google bug?

Sometimes pages drop out of the index because of a problem on Google’s side. In September and October 2020, for example, Google confirmed bugs that removed pages from the index and affected canonicalization, mobile indexing and Top Stories; large news and e-commerce sites lost key pages for days. Before you change anything on your site:

1. Check the Google Search Status Dashboard for ongoing crawling, indexing or serving incidents and ranking updates.
2. Check whether the drop is site-wide or specific: if many unrelated sites report the same issue and your Page indexing report shows no new errors, it’s likely Google’s side.
3. Rule out your own causes using the checklist above (noindex, robots.txt, redirects, server errors).
4. Wait and monitor: after Google fixes a bug, pages usually return without action. Requesting indexing for key URLs in URL Inspection can speed this up.
5. Don’t react hastily to ranking volatility during a core update; compare data after the rollout ends. See why one-off optimization doesn’t last.

FAQ

Why does Search Console say “Crawled – currently not indexed”?

Google crawled the page but decided not to index it, usually because it sees little unique value or similar pages already exist. Improve the content, merge thin pages and add internal links.

What does “Discovered – currently not indexed” mean?

Google knows the URL but hasn’t crawled it yet, often because of many low-value URLs, weak internal linking or a slow server. Prioritize important pages in links and sitemaps.

Does robots.txt remove a page from Google?

No. It blocks crawling, not indexing. To remove a page, allow crawling and add noindex, or return 404/410.

How long does indexing take?

From hours to weeks. Strong internal links, a sitemap and a fast, stable server help; “Request indexing” can speed up individual URLs.

Sources

• Search Console Help: Page indexing report
• Search Console Help: URL Inspection tool
• Google Search Central: Introduction to robots.txt
• Google Search Central: Block indexing with noindex
• Google Search Central: Build and submit a sitemap
• Google Search Central: Consolidate duplicate URLs
• Google Search Central: Redirects
• Google Search Central: JavaScript SEO basics
• Google Search Central: HTTP status codes and network errors
• Search Console Help: Security issues report and Manual actions report