How do search engines work?

Search engines crawl the Web to see what is there. This task is performed by a piece of software, called a crawler or a spider. Spiders follow links from one page to another and index everything they find on their way. Having in mind the number of pages on the Web (many many billions), it is impossible for a spider to visit a site daily. Sometimes crawlers will not visit your site for a month or two, so during this time your SEO efforts will not be rewarded. Beware – there is nothing you can do about this.

Crawlers do not see images, Flash movies, JavaScript, frames, password-protected pages and directories, so if you have tons of these on your site you’d better find out if any of these are crawlable. If they are not viewable, they will not be spidered, not indexed and thus non existant to search engines.

After a page is crawled, the next step is to index its content. The indexed page is stored in a giant database, from where it can later be retrieved. Essentially, the process of indexing is identifying the words and expressions that best describe the page and assigning the page to particular keywords. Sometimes they might not get the meaning of a page right but if you help them by optimising it, it will be easier for them to classify your pages correctly and for you – to get higher rankings.

When a search request comes, the search engine processes it – i.e. it compares the search string in the search request with the indexed pages in the database. Since it is likely that more than one page contains the search string, the search engine starts calculating the relevancy of each of the pages in its index to the search string.

There are various algorithms to calculate relevancy. Each of these algorithms has different relative weights for common factors like keyword density, links, or metatags. Google places a high priority on Links.

The last step in search engines’ activity is retrieving the results and ordering them in terms of relevancy.