Crawling and Indexing

Imagine the World Wide Web as a network of stops in a big city subway system. Each stop is its own unique document (usually a web page, but sometimes a PDF, JPEG, or other file). The search engines need a way to “crawl” the entire city and find all the stops along the way, so they use the best path available: the links between web pages, an example of which is shown in Figure 2-11.

Figure 2-11

In our representation in Figure 2-11, stops such as Embankment, Piccadilly Circus, and Moorgate serve as pages, while the lines connecting them represent the links from those pages to other pages on the Web. Once Google (at the bottom) reaches Embankment, it sees the links pointing to Charing Cross, Westminster, and Temple and can access any of those “pages.”

The link structure of the Web serves to bind together all of the pages that were made public as a result of someone linking to them. Through links, search engines’ automated robots, called crawlers or spiders (hence the illustrations in Figure 2-11), can reach the many billions of interconnected documents.

Once the engines find these pages, their next job is to parse the code from them and store selected pieces of the pages in massive arrays of hard drives, to be recalled when needed in a query. To accomplish the monumental task of holding billions of pages that can be accessed in a fraction of a second, the search engines have constructed massive data centers to deal with all this data.

One key concept in building a search engine is deciding where to begin a crawl of the Web. Although you could theoretically start from many different places on the Web, you would ideally begin your crawl with a trusted set of websites. You can think of a factor in evaluating the trust in your website as the click distance between your website and the most trusted.