How Search Engines Work

In order to understand search engine optimization, it is necessary to understand the basics first. One of the first questions everyone has is: 'What exactly is a search engine?'

Essentially a search engine is a software program that helps the user find information stored on various machines that are connected together via the World Wide Web (the Internet). A search engine allows users to seek content that matches the keywords or key phrases and retrieve a list of websites that match those criteria. One important point to be noted is that the search engines just 'indexes' pages of a website. What this means is that they do not store all the information about the website; that would just be impossible given the number of websites and web pages that exist today. Rather, they just store the 'key words' or 'key phrases'. Once a search engine user logs in a request, it is via the use of this stored information that search engines are able to direct users to the correct website. To keep the information up to date, search engines regularly update their indexes to operate quickly and efficiently. 

Given the enormous amount of websites that are designed, uploaded and updated everyday, it is physically impossible for the search engines to manually check the internet. All search engines use a small code of software called a spider, robot (or just a bot) to index content. These robots or spiders automatically 'crawlâ' a website and store (index) the most important information about it. These robots visit not only all the pages of a website, but also follow the links that are provided from within these pages - that is why they are called as 'spiders'.  Thus, the main objective of search engine spiders is to compile a comprehensive database that delivers the most relevant results to the users' queries.

Although they work in the same way, each search engine company prefers to call its robots or spiders by different names. For example, the robots of Google are known as 'Google bots', those of yahoo as 'slurp' and so on.

There are two basic types of search engines - automated and human powered. The basic difference between them is that in the automated search engines, the indexing is done without any human intervention. In a human powered search engine (also called a website directory), all the submission and indexing is done manually. Google is an automated search engine while yahoo is a website directory (though recently yahoo has changed modes and uses both manual and automated submission).

Irrespective of the way a search engine functions, it is important to understand how they index the website; this is one of the key factors for implementing proper search engine optimization for your company website.

There are other types of search engines that are hybrid, and still others that called 'meta crawlers' that parse the major search engines themselves and returns pages from them in response to a query.