Website Indexing – Make Sure You Are on the Radar

In an earlier post (Page Rank Checker), I illustrated Google’s Page Rank Checker. However, Page Rank and ranking high in the search engines are not the same thing, although Page Rank is one of the 200 factors contributing to Google’s search algorithm (‘correlation not causation’ in the words of the SEOs).

Website indexing is as important to a website as a roadmap is for a journey. Without being indexed, a page will remain invisible to spiders and not show up in search engine results pages (SERPS).

It’s easy to be lulled into a false sense of security because your home page is indexed and therefore not realise that your site’s inner pages might be online wallflowers.

So let’s use Google’s indexing tool to reveal which of our pages are indexed.

Why Google?

Of course there are other search engines yapping at Google’s feet, such as Microsoft’s Bing and Yahoo (Microsoft now also owns Yahoo). But in the UK, and even in the US, Google still has the lion’s share of the market as the pie chart shows.

Source: StatCounter GlobalStats , October 2013

Crawling vs Indexing

How do you get pages indexed?

‘Crawling’ is simply the process that leads to indexing, so that the term ‘indexed’ is used to mean ‘crawled and indexed’.

Here’s how to do it.

Log into Google and type site:your website url. All the individual pages Google has indexed will be shown in the search results. Google Webmaster Tools flags a Crawl Errors page that provides details about the urls in your site that Google could not successfully crawl.

Let’s say the news is bad – you have a 20-page website and find that only your homepage is indexed.

How do you make sure that the rest of your site is visible?

Links and sitemaps are key actions to getting your new pages crawled and indexed. Here are 10 top tips:

1. First rule out anything that may be blocking indexing. A slow loading page is one factor. Broken links is another, as are the use of Flash, JavaScript, audio or video files or text included in images. Frames, duplicate content or inadvertent guidelines to the search engine crawlers not to index a page (eg robots.txt) are others.
2. Use a 301 redirect to make sure Google knows that variations of your site’s name are redirected to your preferred url. This tells Google always to look for your content in one location. Think of it as a postal redirect service.
3. Good internal link structure. Googlebot uses links to crawl from page to page, so make sure your pages are internally linked. Build deep internal links within your site to the pages you want indexed. Make sure you do this naturally though. Link to each page once, and more frequently to the more important pages.
4. Sitemaps guide Google’s spiders to the inner pages of a site. It lists the pages on your website. Use a free xml sitemap generator such as www.xml-sitemaps.com. Then submit it using Google Webmaster Tools – a free set of tools for anyone with a Google account.
5. Resubmit your updated sitemap whenever you create new content on a site or blog. This is the online equivalent of blowing a whistle. WordPress pings automatically. Otherwise use Webmaster Tools (for both Google and Bing). The Googlebot will then re-crawl your site.
6. Good navigation structure is important both for your users and for the fast indexing of your site. Breadcrumb navigation (which shows how you have drilled down the levels, eg Books>Crime, Thrillers & Mystery>Thrillers>legal thrillers) helps clarify the structure.
7. Submit your url to search engines such as Google and Bing, Yahoo being covered by Bing (see Hobo’s How to Submit a Site to Search Engines like Google Yahoo & Bing from October 2013).
8. Inbound links – the Googlebot is more likely to find and index your site when websites that are often crawled and indexed link to it. Build quality links to your site from popular and relevant sites.
9. Offsite content – a tactic that used to be successful was to submit an article to an article bank or the comment section of a blog with a link back to your own website. However, these days many article banks and blogs include ‘no follow’ links, basically a notice to the search engines not to count the link. So check whether any ‘link juice’ is passed on first.
10. A shortcut to getting pages indexed. According to Moz, posting on Google+ offers a fast track to indexing, ahead of Facebook, particularly if the post reader hits the +1 button (showing appreciation). On my list of things to try!