How Search Works

  • Callum Carlström, Connor Keith, Matthew Wimberly

Discover how search engines operate. Leveraging the basics will help you understand how SEO factors into your website’s ranking..

The 21CM Guide to SEO

What search engines want

Let’s start by asking, what do search engines want? The answer, in simple, is traffic. They want people to use them, because then they can monetize that traffic by selling ads. 

To get people to use them – and use them they do; there are more than six BILLION searches EACH DAY – the search engines need to provide us what we want to see. And that means they have to (1) understand what we’re looking for, and (2) know which sites (pages, really) are going to show us what we want to see. So let’s take a look at how they do that.

Search engine structure

The structure of search engines is broken into 3 main elements:

  • Web Crawlers
  • Indexes (Catalogs)
  • Search Engine Software (Algorithms)

WEB CRAWLERS (SPIDERS)

“Spiders” travel (“crawl”) across the internet, inspecting websites and all content on websites. They read and trace the content of a site and all links on a domain to create site maps. These maps are stored for later indexing and updating of websites to maintain databases that search engines look through for results.

Crawlers are capable of reading all front-facing code (HTML, images, CSS, and JavaScript) – that’s what you see when you visit a site – to rank and categorize the sites that they observe. Through reading each site, the crawler can determine headings, keyword patterns, and a basic understanding of the site’s subject matter.

This information is then compiled and cataloged for later use in algorithmic searching. These catalogs are updated frequently by crawlers to give accurate search results.

Web Crawler graphic

Indexes (Catalogs)

The information gathered by web crawlers is stored and indexed in a catalog. Without this index, the search results would take a substantial amount of time to create because the “Spider” would have to go to each website to catalog it prior to showing the search result.

The index contains all websites that have been crawled by spiders. This information is categorized into databases that can be queried by search algorithms to provide results for the user.

a diagram of a database

SEARCH ALGORITHMS

Each search engine has its own unique algorithm (formula) that it uses to rank and display pages. Each page is given a rank based on the search engine’s calculation of the relevancy of that page to the consumer’s search parameters. Each search engine uses a different algorithm or method for calculating the ranking of each result. The exact algorithms are closely-held secrets, but the results are primarily derived from the following:

  • Location and frequency of keywords (and key phrases) on a web page
    • Keywords in title tags are considered more relevant
    • Keywords are found in the bodies of text (patterns of use are more relevant)
  • Off-page ranking
    • Link analysis – How pages link together and how much “authority” each has
      • Artificial link spamming is discounted (i.e., paid links)
    • Click through measurement (transaction rate)
a diagram of a database

Summary

Here’s a quick recap of how search works, the video is short but covers the important facts we discussed above

SEO Guide Home
Keywords