How Search Really Works
How Search Engines Crawl Index and Rank
The three-stage pipeline every page must pass through — crawling, indexing and ranking — and the common ways pages fail at each stage.
Reading time · unlocks the next lesson
0:00 / 5:00 · pausedCrawling: discovery
Crawling is Googlebot following links (and sitemap entries) from page to page, downloading HTML to see what's there. A page that has no internal or external links pointing to it, or that is blocked by robots.txt, may never be crawled at all. Crawl budget — the number of pages Googlebot is willing to fetch from a site in a given period — matters more on large sites than small ones, but wasted budget on thin or duplicate pages is a problem at any scale.
New pages are typically discovered fastest through an XML sitemap and internal links from already-indexed pages; relying solely on Google 'finding' an orphan page can take weeks.
Indexing: storage and understanding
Once crawled, Google renders the page (running JavaScript where needed), extracts the text, and decides whether to add it to the index — the enormous database search results are drawn from. A page can be crawled but excluded from the index if it's marked noindex, is a near-duplicate of another page, or is judged too low-value to store.
Check indexing status directly in Search Console's URL Inspection tool rather than guessing from a site: search, which is an unreliable proxy for what's actually indexed.
- Crawled ≠ indexed — both stages can fail independently
- noindex tags, canonical tags and robots.txt each block indexing differently
- JavaScript-heavy pages need server-side or pre-rendering to be read reliably
- Thin or duplicate pages are the most common reason for exclusion
Ranking: the competition
Ranking only happens among indexed pages: Google compares hundreds of signals — relevance, content quality, links, page experience, user behaviour — to order results for a specific query. Ranking is always relative to what else is indexed for that query, so a page can improve in absolute quality yet still fall in position if competitors improve faster.
This is why SEO work is never really 'finished' — it's a moving target against every competing page targeting the same intent.
Diagnosing pipeline failures in RankAudit
RankAudit's crawl report flags each stage separately: fetch errors and blocked resources point to crawling problems, noindex/canonical conflicts point to indexing problems, and thin-content or missing-signal warnings point to ranking problems. Work through the report top to bottom rather than jumping straight to ranking fixes on a page that isn't even indexed yet.
A quick habit: whenever a page underperforms, check its crawl status before touching its copy — you may be optimising a page Google can't even see.
Key takeaways
- ✓Separate crawling, indexing and ranking as distinct failure points
- ✓Use Search Console's URL Inspection tool, not site: search, to confirm indexing
- ✓Fix crawl and index issues before worrying about ranking position
- ✓Re-run RankAudit after fixes to confirm the pipeline is clear
Why this lesson matters
This lesson belongs to How Search Really Works, the part of SEO Foundations where the goal is: understand what seo is and how search engines actually find, index and rank pages.
Read it once, then do it straight away on a real site inside RankAudit. Nothing here is theory for its own sake — every step produces something you can show a client.
Do it now
- 1Open RankAudit with sample data already loaded, so you are not stuck on setup.
- 2Crawl a site and triage the issue list.
- 3Open RankAudit and run its Crawl Diagnostics on your homepage to see how a search engine bot 'sees' your site.