Google and other search engine working

How Search Engines Work

Many people use the term “search engine” to describe a variety of directory sites that have different purposes and inner workings. Let’s review those briefly, to cover the distinctions between them. There are three major types of sites which are generically called “search engines:” search engines, directories, and hybrid search engines.

Search Engines: True search engines crawl the web, then automatically generate their listings. If you change your web
pages, search engine crawlers will eventually find these changes, and that can affect your listing. Page titles, body copy,
meta tags (sometimes) and other elements all play a role in how each search engine evaluates the relevancy of your page
(and hence its ranking). There are plenty of ways that you can cater to a search engine’s crawlers and change your site to
help improve its rankings.
A search engine has three “parts:” first is the crawler, or spider. The spider visits web pages, reads them, follows links
into the site and transmits what it finds back to the search engine’s master database, or index.
The index (or catalog) contains a copy of every web page found by its spiders/crawlers. If a crawler sends back
new/updated information on a page, it updates the page’s entry in the index. The third part is the search engine software, which is the engine’s user interface. The software sifts through the index in response to user searches and returns a
listing of sites, based on what it finds most relevant.
Directories: A directory (such as Yahoo!) works only by having sites submitted to it by humans. You enter a short
description of your site, or the directory’s editors write reviews of sites that attract attention. That description then stays
there until you resubmit your site; just changing your site will have no effect on how the site is listed.
Furthermore, any search on your site in a directory will bring up only those keywords you chose to identify it with when
you submitted it (or the editors did when they reviewed your site). Normal “search engine” tips for improving your site’s
ranking will have no effect on a directory’s listing (although a well-designed site might make it a better candidate for a
review).
Note, however, that Yahoo! is unique from all other directories in that you could be listed on Yahoo! without having submitted your URL. They make use of a robot that searches for new sites at certain Internet locations (i.e. announcement
sites, etc.). Despite Yahoo!’s implementation of this search robot, mostly all of their new additions still come via their
“Add URL” form.
Hybrid Search Engines: Some search engines maintain a directory that “works with” the information its crawlers
retrieve. Being included in one of these directories is as much luck as it is skill, since you can submit your site, but there’s
no guarantee of it being included.

How Searches Work

Each search engine works differently. There is no standard way to determine search string relevance in a search. Some
engines determine relevance only by the user-visible text on the page (Lycos). Some only use keywords (Excite). Some
use a combination (AltaVista/InfoSeek). There are also intelligent agents out there that use who knows what to index
on.
Several different factors influence how a search results page is built. These factors include:
• what word is being searched for
• whether Web pages containing the word were ever registered with the engine or spidered by the engine.
• whether meta keywords are used or ignored
• whether the contents of the Web page are processed or ignored
• how the particular search algorithm weighs the different aspects of a Web page in determining the relevancy of any
word being search on.