CRAWL BUDGET

Google allocates each site a crawl rate limit — a maximum number of requests Googlebot will make to your server within a given time period. When Googlebot exhausts this allocation, it stops crawling. It does not pause and resume where it left off. The session ends and the next starts from the beginning of the prioritized crawl queue.

The prioritized crawl queue is not random. Google assigns crawl priority based on page authority (internal and external links), freshness signals (how often the page changes), and perceived user value (clicks and engagement). High-priority pages are crawled frequently. Low-priority pages are crawled infrequently or not at all within any given session.

When crawl budget is tight, the budget gets consumed by low-priority pages before Googlebot reaches high-value content. New articles go unindexed for days or weeks. Updated pages retain stale cached versions in the index. Price changes, content revisions, and new sections are invisible to Google until the next crawl.

The indexing consequence is visible in Google Search Console under Coverage. Look for "Crawled — currently not indexed" and "Discovered — currently not indexed." On sites with crawl budget problems, these categories grow over time — a growing list of content invisible to search despite being published.

Crawl budget problems are self-reinforcing.

Phase one: budget depletion begins. Your site accumulates low-value URLs — faceted navigation combinations, parameter variants, duplicate content from session IDs, soft 404 pages. Each one consumes budget previously allocated to important pages.

Phase two: crawl frequency drops for high-value pages. Key landing pages that were crawled daily may now be crawled weekly. New content takes longer to appear in the index.

Phase three: rankings drift on stale content. Updated pricing pages, new article sections, and revised content are invisible to Google. Rankings are based on outdated versions.

Phase four: engagement signals decline. Users see current content but arrived based on cached versions. Mismatches increase bounce rates, further reducing perceived page value to Google — accelerating deprioritization.

Phase five: the vicious cycle. Reduced crawl frequency creates stale content, which creates rankings drift, which creates reduced traffic, which creates reduced engagement signals, which creates further reduced crawl priority.

For e-commerce sites: check GSC Coverage for submitted vs indexed URL counts. If 50,000 pages are submitted but only 12,000 indexed, crawl budget is being consumed before important product pages are reached. Use Screaming Frog to identify URL parameter patterns and how many unique URLs each generates.

For content sites: the primary suspects are category pages, tag archives, author pages, date archives, and paginated series. WordPress alone generates dozens of low-value URL patterns per post. An 800-post blog may have 2,000-3,000 indexable but low-value archive URLs.

For any site: use server access logs to see what Googlebot is actually crawling. Access logs provide ground truth no third-party tool matches — exact URLs, order, frequency, and server responses. Compare crawl frequency of important pages against low-value pages.

The key diagnostic question: does the number of URLs Googlebot could theoretically crawl exceed your daily crawl allocation? If your site generates 500,000 crawlable URLs but your daily allocation is 10,000 requests, Googlebot will never reach the bottom of the queue.

Fix 1: noindex low-value URL types. Tag archives, author pages, date archives on content sites. Parameter-based variants on e-commerce sites. Noindex removes them from the crawl queue over time.

Fix 2: block parameter-generated URLs in robots.txt. For faceted navigation parameters (color, size, sort, filter), block the pattern using Disallow. This prevents Googlebot from following these URLs entirely — the most aggressive and fastest crawl budget recovery.

Fix 3: fix redirect chains. Every hop consumes a crawl request. A 301 chain using three requests reaches the same destination as a direct redirect using one. Audit and collapse all chains to single hops.

Fix 4: clean your XML sitemap. Include only canonical, indexable, high-value URLs. Remove noindex pages, redirecting pages, and low-value archives. A focused sitemap steers Googlebot toward important content.

Fix 5: improve server response time. Faster responses allow more requests within the same window. Reducing response time from 800ms to 300ms can increase effective crawl budget by 30-50%.

Fix 6: strengthen internal linking to important pages. Pages with more internal links signal higher crawl priority. Build comprehensive internal links from homepage, hub pages, and popular content to pages needing frequent crawling.