How to Get Google to Index Your Site With the Coverage Report

If you want to call yourself a technical SEO, you need to be using the Google Search Console Index Coverage report.

It’s an invaluable tool for understanding:

Which URLs have been crawled and indexed by Google and which have not.
And, more importantly, why the search engine has made that choice about a URL.

The report is easy to follow because it uses a traffic light color scheme.

Red (Error): Stop! The pages are not indexed.
Yellow (Valid with warnings): If you have time to spare, stop, otherwise, hit the gas and go! The pages may be indexed.
Green (Valid): All is well. The pages are indexed.

The problem is, there’s a big grey zone (Excluded).

The road rules may as well be written in a different language when you try to read them: Googlish! So today, we will translate the status types in the Index Coverage Report into SEO action items you should take to improve indexing and drive up organic performance.

What is the Google Index Coverage Fixing Report?

The Google Index CoverageFixing Report provides information about the indexing status of all URLs that GoogleBot visited or attempted to visit in a specified Google Search Console property. The results page for each URL shows the data for that URL only. The errors, warnings, and valid statuses are all grouped together. The status code also provides the reason for that status, which is especially helpful for not found (404) errors.

The Google Index Coverage Fixing Report will soon be available as an experimental feature to a select group of beta users.

How do you know if your mobile site is in Google’s index?

You can view a live URL of your site in Search Console. Test whether an AMP page on your site is able to be indexed by doing the following:

Navigate to the corresponding property in your GSC.
Open up the Google URL Inspection tool.
Cut and paste in a specific URL’s current index status.
Look to see if it says “Coverage – Submitted and indexed”.
Below that check if it says “Linked AMP version is valid”.
You can both “TEST LIVE URL” and “REQUEST INDEXING”.
You can also use Google’s AMP page validation tool

Whether you are a business that serves a B2B audience or B2C clients, it is important to manage how your pages get indexed and crawled by search engines, in order to survive online.

SEO Impacting Issues in the Index Coverage Report

Don’t focus only on fixing only the errors. The bigger SEO wins are generally found in the areas that are not excluded.

The following are the most important Index Coverage report issues for SEO, listed in order of importance, so you can prioritize where to focus your attention.

Discovered – Currently Not Indexed

The reason the URL is not appearing in search results is that Google has not crawled the site yet. This is often due to the site being linked to or having an XML sitemap. This indicates a crawl budget issue.

If there are only a few pages that need to be fixed, manually trigger a crawl by submitting the URLs in Google Search Console.

If there are a lot of errors, spend time fixing the website architecture (including URL structure, site taxonomy, and internal linking) to solve the problems with crawling.

Crawled – Currently Not Indexed

The reason Googlebot did not index the URL is that it found the content not worthy to be included. The most common reason for a website to be demoted in ranking is due to problems with the quality of the website’s content, such as thin or outdated content, or because the website is full of spammy user-generated content. If the content you’re trying to index is good but not appearing in search results, it’s probably due to rendering issues.

– Go over the content of the page.

If you understand why Googlebot has not indexed the page’s content, then ask yourself a second question. Does this page need to exist on my website?

If the answer to the question is no, then the URL should be 301 or 410. You should add a no index tag to your website if there is an issue with the content on your website that needs to be fixed. You can prevent a page from being crawled if it is a parameter-based URL by following best practices for parameter handling.

If the content quality appears to be good, check to see what renders without JavaScript. Google is capable of indexing JavaScript-generated content, but it is a more complex process than HTML because there are two waves of indexing whenever JavaScript is involved.

The first wave of indexing pages is based on the initial HTML from the server. This is what you would see if you were to right-click and view the page source.

It is full-text, meaning that it indexes the contents of all fields, not just those designated as “searchable.” The second indexes is based on the DOM, which includes both the HTML and the rendered JavaScript from the client-side. It is full-text, meaning that it indexes the contents of all fields, not just those designated as “searchable.” This is what you would see if you were to right-click and inspect an element.

The challenge is that Google doesn’t index all the content on a website immediately. They wait until they have the resources available to do a more thorough job. This means that content that relies on JavaScript takes longer to index than content that only uses HTML. A few weeks at most from when it was first crawled.

If you want to avoid having your content indexed slowly, use server-side rendering to ensure that all your key content is included in the initial HTML file. To optimize your website for search engines, you should include page titles, headings, canonicals, structured data, main content, and links.

Duplicate Without User-Selected Canonical

The page is considered by Google to be duplicate content because it is not marked with a clear canonical. Google does not think this page is important and has removed it from the search results.

You can fix this issue by including a rel=canonical link on every crawlable URL on your website. This will explicitly mark the correct canonical URL for each page. The canonical URL for a given page can be found by inspecting the URL in Google Search Console.

Duplicate, Submitted URL Not Selected as Canonical

The URL was explicitly requested to be indexed, for example by submitting it in an XML sitemap.

How to fix it: For every URL on your website that can be crawled by search engines, use a rel=canonical link to explicitly mark the correct canonical page. Then, include only canonical pages in your XML sitemap.

Duplicate, Google Chose Different Canonical Than User

The page has a rel=canonical link in place, but Google thinks a different URL should be the canonical.

Look at the URL to see which one Google has selected as the canonical URL. If you agree with Google, change the rel=canonical link. If not, work on your website architecture to reduce the amount of duplicate content and send stronger ranking signals to the page you want to be the canonical.

Submitted URL Not Found (404)

The URL that you submitted as part of your XML sitemap does not exist.

To fix this, you can either create the URL that is referenced in your XML sitemap, or remove it from the sitemap. This error can be avoided by following the best practice of dynamic XML sitemaps.

Redirect Error

Cause: Googlebot took issue with the redirect. The most common cause of this problem is a redirect chain that has five or more URLs. Other causes include redirect loops, an empty URL, or an excessively long URL.

If your website is not redirecting properly, you can use a debugging tool like Lighthouse or a status code tool like httpstatus.io to figure out what is breaking the redirect and how to fix it.

The 301 redirects should always point to the final destination, and if this means editing old redirects, then that is what should be done.

Server Error (5xx)

Servers are unable to load a page when they return a 500 HTTP response code, also known as an Internal Server Error. The reason that a website is not appearing in Google search results could be due to general server problems, but more likely it is because the server was briefly disconnected, preventing the Googlebot from crawling the site.

If it’s something that only happens every once in a while, don’t worry about it. The error will resolve itself after some time. If a page is important, you can tell Google’s search bot, called Googlebot, to visit the URL by requesting indexing within the URL inspection feature. If the error keeps happening, talk to your system engineer / tech lead / hosting company to improve the server infrastructure.

Crawl Anomaly

The URL was not crawled because something prevented it, but even Google does not know what that something is.

If this is a new page and you still see these error codes, that means that there’s an issue with the URL itself. If you are seeing 4xx or 5xx level response codes when you use the URL Inspection tool, that means there is an issue with the URL itself. If the first step does not provide any clues, send the URLs to your development team.

The Impact that Not Getting Pages Indexed Has on Business

You cannot build your site credibility if your web pages are not indexed.

We build business relationships differently. The way we form new connections and maintain relationships has changed because of the internet. We now have the ability to connect with people we wouldn’t have been able to otherwise, including business partners and celebrities. Since people can communicate online without ever meeting in person, it’s important for businesses to know how people interact with online content. Doing so can help businesses to grow. You have the opportunity to increase your visibility by appearing in new product carousels on SERPs.

Google wants to help you improve your code so that your pages will be indexed and crawled more easily. This means that users who are searching for related content are more likely to find and consume the content that has been indexed.

Google probably pays attention to your reviews and product data when it indexes your web page. This feature may improve over time after gathering feedback from users. This new Search Console function from Google is designed to speed up the process of re-indexing pages with fixed content. If your page has been negatively impacted by schema errors, it may be beneficial to improve your markup before the site is indexed again. Make sure to tell Google that you have taken care of the problem with your website’s content or coding.

Monitoring your site’s indexing errors is important as a large number of errors could send a signal to Google that your site is poorly managed or has a low health score. Many small businesses do not have the resources to invest in ongoing site maintenance and ignore their error reports or forget to mark errors as fixed. Seeing a long list of errors can be overwhelming. It’s important to clean your list of indexing errors regularly and to set aside some time to watch for new errors and clean them up as they occur. To ensure that healthcare pages providing essential patient services are indexed correctly, extra steps must be taken.

What Benefits Do SEOs Gain from the new Indexing Workflow?

The ability to improve your SEO more quickly and easily.

If you want to prevent search engines from indexing pages that have low or no SEO value, you can check to make sure they’re not indexed in the visual report. If you have already set up URL parameters in Google Search Console so that Google will not crawl or index the same pages with different parameters separately, you should mention this.

If your CSS files are causing problems that prevent a page from indexing, Google wouldn’t be able to index the pages the way you want them to. The provided examples are meant to reduce diagnostic time.

Another common problem is if your JS isn’t able to be crawled. Google will be able to index your site’s dynamically generated content better with improved error reporting and quick fixes.

* Submit your sitemap right there.

To quickly study individual sitemaps, filter your Index Coverage data.

While you should index your site for all major search engines, this new Search Console indexing error report will help you to be better indexed across them all.

How the new Search Console Indexing Reports Look and Function

Web developers and SEO specialists are looking forward to investigating these new reports and using them to improve websites. Glen Gabe, President of G-Squared Interactive LLC, released several screenshots in a LinkedIn article on September 7, 2017, from the new Index Coverage Report in Google Search Console* (GSC).

The new Google Index Coverage report will show you which of your pages are indexed, even if you have not submitted them to sitemaps. The app does not seem to have an export feature. If you want to know where you appear in the “People Also Ask” boxes on Google’s search engine results pages (SERPs), this information can be helpful.

Glen commented that the site was indexed but blocked by robots.txt. The URLs are actually 404, but Google can’t crawl them to know.