Xavier Media®

This is a guest post and the views of the author do not necessarily reflect the views and opinions of this site. The author, Brian Nixon, is a professional search engine optimizer who writes for Pitstop Media Inc, a Canadian company that provides top rated SEO services to businesses across North America. For more information please visit www.pitstopmedia.com.

Sometimes we forget to start with the basics and we jump on the SEO bandwagon to implement intermediate or advanced techniques without really understanding why we do it. How does a search engine works at its back end? What patents, technologies are involved? Are you familiar with the terms database barrel or DumpLexicon? No worries, probably not many know. Speak with your SEO company, see if they know about it. If they know at least a bit, then you’re in good hands.

How Google Works?

Google's is undoubtedly one of the best and the most powerful search engine. The company's efforts and time spent in building an efficient search tool are clearly evident from its success. Like any search engine Google has its own algorithm for generating the search results. The crawlers / spiders and index of keywords are crucial elements which play a key role in getting the most relevant results.

Besides how keywords rank in search results, the SERP or search engine results page is something that differentiates Google from other search engines. To assign relevance score to web pages, Google uses its trademarked algorithm, known as Page Rank shorthanded, PR.

The Page Rank of web pages depends on a few important factors:

1. Location and frequency of words
   The density and placing of keywords need to be controlled. If the target keyword appears only once in the entire article that particular keyword will receive a low score.
2. Age of the document
   As people create new web pages on a daily basis, not all are able to stick around for a long time. Web pages with an established history get more preference and value from Google.
3. Backlinks to from other sites
   While crawling through the web pages on the internet, Google also looks at the number of web pages linking to a particular site, but it also looks at the quality. This helps Google to determine its relevance.

Search results display process

After you entered you search query in Google's search box, it takes less than a second for the search engine to generate and display the results on screen. Here it is interesting to know how Google finds relevant pages and determines the order of websites appearing in the search results.

Before the web became a visible part of the internet, there were a few search engines to look for information. Programs like Archie and Gopher played a significant role in indexing all files that are stored in internet servers. Using these programs, the amount of time needed for finding documents and programs was dramatically reduced. Let’s learn more about the process.

Some metrics used by Google to rank sites.

Web Crawling

Before a search engine can locate a document or a file, it has to be indexed first. To look for any particular information on millions of web pages, the search engines have special software robots, also known as spiders. These spiders build a list of all the words that are found on

websites. This process is called as web crawling. While this seems a simple, the search engine spiders have to actually go through all the web pages that are live on the internet.

One may wonder how the search engines index these pages. To begin with, the Googlebot makes use of a huge set of computers to fetch web pages. Here, Googlebot is the program that facilitates the fetching process. What’s more, the Googlebot also determines the sites that are to be crawled, the number of pages to be crawled from each site and the frequency of crawling.

Google Indexer

When Googlebot starts to crawl the internet, it begins with a list of URL's from the earlier sessions. This is augmented with the Sitemap data offered by webmasters. The moment Googlebot lands on a page, links from those pages are taken and added to Google's list of pages to be crawled. New updates to websites are regularly updated to Google index.

To compile the index of all words, Googlebot processes all pages that it crawls through. In this process, Googlebot also checks the location of these words in each page. Additionally, Google also process the information that is included in content tags, like ALT attributes and title elements. Each document is converted into a set of word occurrences called hits, which are then distributed into “barrels”.

Here, it is important to know that Googlebot cannot process all types of content. For instance, Googlebot is unable to process iframes or dynamic pages, rich media files, etc.

Google's index is sorted in an alphabetic order by the search term. Each index entry stores a list of documents, along with search terms and the location. To enhance the search performance, Google does not index common words (also known as stop words) like on, or, the, why, how, single letters and digits. While multiple spaces and punctuations are ignored by the indexer, to improve Google’s performance, all letters are converted to lowercase.

Serving Results (search query processor)

This is where the actual work is done. It’s not an easy process and involves computer algorithm. The real working procedure of 'search query processor' is not revealed by any of the search engine providers. However, SEO professionals have enough knowledge on how to improve the positioning of your website in Google's search results. With appropriate keyword density and page rank, SEO professionals can help you get your web page on top search engine results.

When you enter a query in Google's search box, spiders crawl through web pages to look for matching pages in the index. The most relevant results are then displayed in the search results. There are more than 200 factors that determine the relevancy of your search query and the content available on web pages. One of these factors includes a Page Rank for any given page. It determines the importance of a web page based on backlinks from other pages.

The popularity of a web page, the size and position of search terms appearing within the page and the relevancy of search terms compared to other page are other things that play a crucial role in displaying accurate search results. To improve its performance, Google applies some machine-learning techniques, which helps the Googlebot and spiders to learn the association and relationship within the stored data.

How to improve crawling and indexing of your site?

Understanding this back end technology is useful but to improve Google indexing and rank of your site, it is important to ensure that Googlebot crawls and indexes your site appropriately. If there are dead or broken links on your site, it will create a negative impact on the ranking of your site. Using Google webmaster tools, you can ensure that your site is crawled by the search engine. To improve the ranking of your website, it should comply with the guidelines set by Google.

Once you understand the basics of search engine technologies, start by creating a testing ideology and take small steps. Arm yourself with lots of patience too. Good luck!

About the author
The author, Brian Nixon, is a professional search engine optimizer who writes for Pitstop Media Inc, a Canadian company that provides top rated SEO services to businesses across North America. For more information please visit www.pitstopmedia.com.