Getting a Crawl Running
Recrawl & Progress Tracking: Measure Your Fixes
Schedule recurring crawls and compare results over time so you can measure whether fixes are actually reducing issue counts and improving the health score.
Want to earn a Rankar Academy certificate?
This guide is free to read in full, with nothing to sign up for. Join the Academy to track your learning, complete courses and sit the certification assessment.
Join the Academy →Performance model
RankAudit · Getting a Crawl Running
Stage 1
Scheduling recurring crawls
Stage 2
Manual recrawl and partial recrawl
Stage 3
progress history chart
Scheduling recurring crawls
In project settings, the Recrawl tab lets you set a frequency — daily, weekly or monthly — and a preferred time window. Scheduled crawls run automatically and append to the project's history without needing manual intervention each time.
- Set frequency: daily, weekly, monthly
- Choose a time window to avoid peak traffic
Manual recrawl and partial recrawl
Beyond the schedule, you can trigger a manual recrawl at any point, or run a partial recrawl limited to specific URL paths, which is faster when you only need to confirm a fix on a subset of pages after a deployment.
- Manual recrawl for on-demand checks
- Partial recrawl to verify a specific URL path
The progress history chart
The Progress tab plots health score and issue count across every historical crawl on a single timeline. Hovering any point shows the exact numbers for that date, and you can click through to open the full report snapshot from that crawl.
Comparing two crawls side by side
Selecting two dates from the history opens a comparison view highlighting which issues appeared, which disappeared, and which pages changed status, making it straightforward to report progress to stakeholders.
Why this matters
Neglecting regular crawl comparisons, or misinterpreting their data, can have significant adverse effects on your SEO strategy. Imagine a scenario where you've implemented a large-scale internal linking overhaul, expecting a boost in crawl depth and indexation for critical pages. Without consistent recrawls and meticulous progress tracking using RankAudit's comparison features, you might miss that half your new links are canonicalised away, or that a deployment introduced a new JavaScript rendering issue, rendering your improvements moot. This leads to wasted development effort and persistent underperformance.
Conversely, diligently leveraging RankAudit's recrawl and progress tracking capabilities empowers you to validate your optimisations with precision. Consider fixing 404 errors across hundreds of pages following a site migration. By scheduling a recrawl immediately after the fix and comparing the 'broken links' report, you can verify the issue count has dropped to zero. This empirical evidence allows you to confidently move on to the next task or, if issues persist, quickly identify regressions, preventing them from accumulating into larger, more damaging problems that impact user experience and search engine visibility.
Prioritising Issues from Recrawl Deltas
When a recrawl completes, RankAudit will highlight changes, but not all changes carry the same SEO weight. Your focus should be on 'critical regressions' – new high-severity issues appearing since the last crawl – and 'persistent critical issues' – high-severity issues that haven't decreased despite implementation efforts. For example, a sudden increase in 'pages with no index tag' is far more urgent than a slight rise in 'pages with missing H2s' if you haven't explicitly aimed to fix the latter. Use the comparison view to filter specifically for new issues flagged as 'Error' or 'Warning'.
Conversely, don't be immediately alarmed by small increases in 'informational' issues, such as a slight rise in pages with long titles, unless this directly contradicts a recent optimisation effort. The true value lies in assessing the trend for specific issue types you've been actively working to resolve. A consistent decrease in 'broken internal links' or an increase in 'pages with valid canonical tags' across multiple crawls confirms your fixes are effective and sticky. Develop a heuristic for what constitutes an 'acceptable' vs. 'actionable' delta based on your site's size and recent changes.
When prioritising deltas:
Focus on new 'Error' or 'Warning' issues first.
Verify decreases in critical issues you've targeted.
Ignore minor increases in 'Notice' level issues initially.
Investigate significant increases in previously stable metrics.
Do it now
Immediately after a new crawl finishes, navigate to your site's dashboard in RankAudit and locate the 'Crawl History' section. Here, you'll find a list of all completed crawls for your site. Select the most recent crawl and one preceding it, then use the 'Compare' option to initiate a side-by-side analysis. Pay close attention to the 'Health Score' trend and the 'Issue Categories' to identify any significant changes between these two points, confirming either improvement or regression.
- Go to your site's dashboard in RankAudit.
- Find the 'Crawl History' widget.
- Tick the boxes for the two most recent crawls.
- Click the 'Compare' button to view their differences.
Key takeaways
- ✓Schedule recurring crawls so trend data builds automatically
- ✓Use partial recrawls to quickly verify a single fix
- ✓Review the progress chart before and after major site changes
- ✓Compare two specific crawl dates when reporting results
Do it now
Set up your first site audit and learn to read what it produces. Crawl a site and triage the issue list.