Everyone relies on Google to some extent. The process of showing up on search engines goes something like; Make some content, optimise said content, publish said content, wait for Google to index said content. The first 3 steps of that process are completely within your control, but unless you’ve got access to some secret Google server room, that last step is entirely out of your hands.
All websites, especially the larger ones, will have a number of pages that aren’t currently indexed, and that’s okay. There are a variety of pages and tags that won’t and probably don’t need to be indexed. However, you do want your main content to be indexing.
Pages with javascript might make Google’s site crawlers a bit more confused than normal, and pages written in HTML, though much simpler to Google crawlers, can also struggle with indexing issues. The truth is that it’s very difficult to know when something is or isn’t going to be indexed. There are some common issues that occur, and steps you can take to mitigate these risks, but sometimes things just happen that are completely out of your control. The first thing to do when this does happen, is to find out why and resolve the problem.
Google Search Console
Search Console is your go to hub for all of your Google related website issues. It’s where you’re going to be running all your diagnostics. There are status reports on there for pages with link errors, pages with 404’s, but we want to look specifically at the pages that aren’t being indexed (despite having no obvious errors).
Normally, Search Console will give you 1 of 2 results in this instance. It doesn’t tell you specifically what’s wrong and where, but it’s where you can start your diagnostics process. The two options generally are:
- Crawled – currently not indexed
- Discovered – currently not indexed
Discovered or Crawled
The more worrying of the two options, would likely be the ‘crawled’ option. This means that Google has actively seen your page, but has decided it doesn’t want it to be indexed. This may be an overall quality issue, with Google deeming that your page doesn’t meet the standards set to rank for your chosen keyword. Try to optimise the page a bit more with some unique headlines, alt tags, images, and copy itself.
The 2nd option, discovered, can mean the exact same thing. However, it’s more likely when you get this message, that Google simply has either too many URLs in its queue, or the whole page itself isn’t linked anywhere else internally. In both instances you should select the request indexing button, but first, make sure that the page itself has the best chance to be discovered beforehand. Try finding other pages of yours that are already ranking well, and link to your new page if you can. Your website pointing towards this new page should help this process resolve itself.
The final thing that is often an issue here is simply that Google thinks this is duplicate content. Perhaps you have a very similar page elsewhere on your site, with similar messaging or similar keywords. Explore some similar keywords within the pages on your site, and see if you think this may be the issue. You could try to explore some slightly different messaging, target a different location or area expertise, or expand on your previous idea.
Crawled and discovered messages do mean that there isn’t any errors on the page. If there are error messages showing up there may be a whole host of other problems. The fundamental indexing issues however, usually arise and can be fixed with simple solutions.
Mitigating Indexing Risks
It’s always useful if you can get ahead of this indexing problem, by avoiding the problem in the content creation phase all together. Here are a few ways you can increase the probability of your page indexing in the first place.
1) Internal linking
This is arguably the main key signal to Google that any given page is important, and you want to recognise it as such. Find pages that are already indexed and ranking well, and try to weave in a link to your new page manually.
2) Eliminate duplicate content
Have a trawl through some of your older blogs, older ultimate guides, any of your published content, and see what topics and conversations you’ve already had. If you want to write some similar pieces, avoid replicating anything that’s already been said before. Otherwise, just find something new and unique, there’s plenty of ideas that you can write about, and new trends, cases, and inspiration that you can find for your content.
3) Optimise each page for SEO
Each page will need its own keywords, images, copy text, alt and title tags, and on-page technical optimisations to be sweet and sound on its own. If you’re not overly familiar with SEO, you can use a very simple plug-in on your website. Something like Yoast SEO, for example. It tells you exactly where to improve, giving your content the best overall chance to index properly.
4) Create a sitemap
Something like Yoast SEO has a setting to enable the default XML sitemap, which creates a pretty solid blueprint of your website for Google’s convenience. If you don’t have any experience with sitemaps, again there are plenty of tools you can use to assist you. Yoast SEO again, Google XML, or Screaming Frog, for example, are three popular choices. Once complete submit your sitemap to Google Search Console, and you may see your content crawled and indexed much more efficiently.
However, it is important to note that each page should be easily found irrespective of its place on your sitemap. Each page that you want indexing needs to have value to any user with a given search query.
5) Encourage inbound links
Backlinking is something that isn’t entirely within your control, but is as valuable as all of the above, if not more. Credible external links are a validation from a third party site that your content and page are good, so you want to try and get as many quality unique inbound links as possible!
6) Crawl Budget
Google allocates certain time and resource allowances to each and every website based on user value. This is called a crawl budget. The larger the site the higher the budget usually is. However, If you post extremely regularly or have a very large site, it is possible that your site just has too many pages for Google to crawl every time it visits. The net result is there are pages that are just stuck in a queue for what feels like forever.
The only way to increase this budget and therefore encourage Google to crawl, and thus index your site more regularly, is to improve the quality of your content. Make it much more valuable to the searches, and you’ll get more visits. More visits will result in Google allocating your site more of it’s crawl budget.
7) Avoid fake 404 signals
A very niche, and unlikely source of indexing error, but ‘soft 404’ signals can cause problems. ‘error 404’ signals are important to avoid as they indicate to Google that your page can’t be found, impacting your SEO. Simply avoid this by trying not to use the words ‘not found’ or the number ‘404’ as this can sometimes trick the algorithm. This blog post will have to be the exception to the rule…
Final Thoughts
While every business relies on Google to a degree, you can take a lot of your own control with your content, your website, and your search engine results. Ensure that when you’re making great content, you focus a percentage of your time on SEO. Eradicate any duplicate content, and get a decent interlinking and backlinking strategy in place. You can even utilise some form of site-mapping or SEO tool to assist you if you aren’t well versed yourself!
Finally, once everything is uploaded, check Google Search Console regularly. Ensure there are no errors first and foremost, and then have a look at your undiscovered, crawled, and non-indexing content. There are always fixes you can make to boost your search engine results, whether it be to your javascript or html coding, or just fixing some incorrect links.
If you’re looking for any more help with Google Search Console and your SEO strategy, please contact us here for a free consultation!
