Many people think search engines have a hidden agenda, or that they just do not like them. This simply is not true. The goal of the search engine is to provide high quality content to people searching the internet.

Search engines with the broadest distribution network sell the most advertising space. As I write this, Yahoo! and Google are considered the search engines with the best relevancy. Their technologies power the bulk of web search.

Crawler (or Spider):

The crawler does just what its name implies. It scours the web following links, updating pages, and adding new pages when it comes across them. Each search engine has periods of deep crawling and periods of shallow crawling. Rapidly changing or highly important documents are more likely to get crawled frequently. The frequency of crawl has no effect on search relevancy; it simply helps the search engines keep fresh content in their index. The best benefit of having a frequently crawled page is that you can get your new sites, pages, or projects crawled quickly by linking to them from a powerful or frequently changing page.

The Index:

The index is where the spider collected data is stored. When you perform a search on a major search engine, you are not searching the web, but the cache of the web provided by that search engine (its index). Search engines organize their content in what is called a ”reverse index.“ It sorts web documents by words in a language. When you search Google and it displays 1-10 out of 143,000 website it means that there are 143,000 web pages which either have the words in your keyword phrase on them, or have inbound links containing the words in the phrase.

Search Interface:

The search algorithm and search interface are used to find the most relevant document in the index based on the user search. First the search engine tries to determine user intent by looking at the words the searcher typed in. The search engine breaks this down via keyword vectors and compares this to their database to find the most relevant results.

In most major search engines a portion of the relevancy calculations are stored ahead of time and some of them are calculated in real time.

Search Algorithm Shifts:

Search engines such as Google and Yahoo! often update their algorithm dozens of times per month. Often times when you see changes in your rankings it is due to an algorithmic shift or something else outside of your control. Usually if you change something on a page it is not reflected in the search results that same day or the next day.

Please click here to contact us for further information on the above matter.