How Search Engines Crawl, Index and Rank

The three stages every page goes through

Before a web page can appear in Google search results, it must pass through three distinct stages. Understanding these stages tells you exactly why some pages rank and others are invisible — and what to fix when yours fall into the second category.

1Crawling — Google finds your pageGooglebot, Google's automated web crawler, follows links from pages it already knows about to discover new pages. It downloads the HTML of every page it visits and stores it for processing. If no other page links to yours, Googlebot may never find it. 2Indexing — Google decides whether to store your pageAfter crawling, Google analyses the page's content to understand what it is about. It then decides whether to add the page to its index — the database of pages it considers eligible to appear in search results. Not every crawled page gets indexed. Thin content, duplicate pages, and technical errors can prevent indexing. 3Ranking — Google orders indexed pages for each queryWhen a user searches, Google retrieves all indexed pages relevant to that query and ranks them using its algorithm — hundreds of signals evaluated in milliseconds. The pages Google judges most relevant and authoritative appear at the top of the results.

🔑 Key Concept Your page cannot rank if it is not indexed. Your page cannot be indexed if it is not crawled. Your page cannot be crawled if Googlebot cannot access it. Fix these in order: access → crawl → index → rank.

How Googlebot crawls the web

Googlebot starts from a list of known URLs and follows hyperlinks from those pages to discover new ones. This is why internal linking matters so much — every internal link is a path Googlebot can follow to reach pages deeper in your site. Pages with no internal links pointing to them are called "orphan pages" and are often missed entirely.

Googlebot visits each page and downloads its HTML. If your page uses JavaScript to load content dynamically, Googlebot must also render the JavaScript — a second, more resource-intensive process that can delay indexing by days or weeks. This is why JavaScript-heavy websites require special technical SEO attention.

Google does not crawl your entire site every day. It allocates a crawl budget to each website — roughly proportional to the site's authority and the freshness of its content. Large sites with millions of pages must manage crawl budget carefully to ensure Google prioritises their most important pages. For most sites under 1,000 pages, crawl budget is rarely a limiting factor.

✅ Practical Tip Submit an XML sitemap to Google Search Console. This tells Googlebot exactly which pages exist on your site and when they were last updated — significantly improving crawl efficiency. RankAudit generates and validates sitemaps automatically.

What Google actually indexes

Indexing is more selective than crawling. Google crawls billions of pages but indexes only a fraction of them. Pages are excluded from the index when:

• They are blocked by robots.txt — your site's file that tells Googlebot which URLs not to crawl
• They have a noindex tag — an HTML meta tag or HTTP header that explicitly instructs Google not to index the page
• They are thin or duplicate content — pages with little original content, or near-identical copies of other pages on your site
• They have crawl errors — the server returns a 404 or 5xx error, or the page redirects in a broken loop
• They are canonicalised away — Google has decided another version of the page is the authoritative one

The Coverage report in Google Search Console shows you exactly which of your pages Google has indexed, which it has crawled but not indexed, and which it has excluded — with the specific reason for each exclusion. This report should be checked monthly for every site you manage.

How Google's ranking algorithm works

Once a page is indexed, it becomes eligible to rank. When a user performs a search, Google's algorithm retrieves all indexed pages that match the query and scores them across hundreds of signals in milliseconds to produce a ranked list.

The most important signals fall into three categories:

Relevance signalsDoes your page answer the query?Keyword presence in title, headings, and body. Topical depth and breadth. Search intent match — does your page format match what users actually want for this query? Authority signalsDoes Google trust your page?Backlinks from other websites — their number, quality, and topical relevance. Domain history and age. Brand signals and mentions across the web. Experience signalsDo users get value from your page?Core Web Vitals — loading speed, visual stability, and interactivity. Mobile-friendliness. Click-through rate from search results. Time users spend on your page. Quality signalsIs your content genuinely good?E-E-A-T: Experience, Expertise, Authoritativeness, Trustworthiness. Original research, first-hand experience, accurate information, clear author credentials.

📊 Data Point According to Google's own documentation, PageRank (the link-based authority signal) remains one of the top three ranking factors after more than 25 years. Backlinks are not dying — low-quality backlinks are dying. High-quality editorial links from topically relevant sites are more valuable than ever.

The role of machine learning — RankBrain and BERT

Google's algorithm is not a fixed set of rules. Since 2015, machine learning models have played an increasingly central role in how Google interprets queries and evaluates pages.

RankBrain is Google's machine learning system for understanding the meaning behind queries — especially novel queries Google has never seen before. It helps Google match pages to queries even when the exact keywords don't appear on the page, by understanding semantic relationships between words and concepts.

BERT (Bidirectional Encoder Representations from Transformers) processes the full context of a query rather than just individual keywords. It understands that "how to treat a dog bite from another dog" has a very different meaning than "how to treat a dog bite from a snake" — and serves different results accordingly.

The practical implication: stuffing keywords into content no longer works. Google understands meaning, not just word frequency. Writing naturally, covering a topic comprehensively, and addressing the full range of questions a searcher might have is now more effective than any keyword density formula.

What this means for your SEO strategy

Understanding crawl → index → rank gives you a diagnostic framework for any ranking problem:

• Page not appearing in Google at all? Start with the Coverage report in Search Console — it may not be indexed
• Page indexed but ranking on page 3 or 4? The issue is relevance or authority — better content or more backlinks
• Page ranking well but not getting clicked? The issue is your title tag and meta description — improve click-through rate
• Page getting traffic but users leaving immediately? The issue is content quality or intent mismatch

🎯 Your Task This Lesson Check which pages Google has indexed on your site Open Google and type site:yourdomain.com (replacing yourdomain.com with your actual domain). The number of results Google returns is an approximation of how many pages are in its index. Then open the Coverage report in Google Search Console (or set it up if you haven't yet — Lesson 5 in Module 05 covers this in full). Make a note of any pages showing as "Excluded" and why. Open Google Search Console ↗ ✓ Lesson Complete — You Now Know ✓The three stages every page must pass through: crawling, indexing, and ranking ✓Why some pages get crawled but not indexed — and the 5 most common reasons ✓The four categories of ranking signals Google uses to score pages ✓How RankBrain and BERT changed SEO from keyword-matching to meaning-matching ✓A diagnostic framework for identifying exactly why any page is not ranking ← PreviousWhat Is SEO? The Complete Beginner's GuideNext →The 3 Pillars: Technical, On-Page and Off-Page SEO

Apply This With the Rankar Toolkit

Every Rankar Academy lesson is built to be put into practice with the Rankar tool suite. Use these tools to apply search engines on your own site — start with RankAudit, then explore the full stack:

• RankWriter — AI SEO content writer for briefs, outlines and full drafts.
• RankTracker — daily rank tracking and SERP monitoring.
• RankAudit — automated technical SEO site audits.
• RankAIO — AI visibility and answer-engine optimisation.
• RankLinks — backlink building, analysis and outreach.
• RankBridge — internal linking and site architecture.
• RankLocal — local SEO, citations and Google Business Profile.
• RankOps — SEO workflow, tasks and client reporting.
• RankLaunch — content planning and editorial calendars.
• RankMarket — the Rankar backlink marketplace.