Crawl Budget for Ecommerce: When It Matters and How to Fix It

Engineer reviewing ecommerce server access logs

Crawl budget is the set of URLs Google can and wants to crawl on your site. For most stores it is not a problem. It becomes one when faceted navigation, parameters and duplicate paths multiply your URLs far past your real catalog, so Googlebot spends its visits on filter combinations while new products and updated prices wait. This guide shows how to tell which situation you are in, how to find the waste, and what to fix first.

Quicktoimpress
quicktoimpress.com
Make Ecommerce SEO Work Harder
Quicktoimpress builds growth platforms and improves SEO and answer-engine visibility for teams managing complex ecommerce technology.
Visit

TL;DR

  • Google defines crawl budget through two things: how much your server can handle (crawl capacity) and how much Google wants your content (crawl demand). (Google)
  • Google says the topic mainly concerns sites with 1 million or more unique pages, sites with 10,000 or more pages that change daily, and sites with many URLs marked “Discovered - currently not indexed.”
  • In ecommerce, the usual culprit is faceted navigation: a catalog of 800 products can look like hundreds of thousands of URLs to a crawler.
  • Fix in this order: shrink the URL inventory, block what has no search value with robots.txt, clean up redirects and soft 404s, keep sitemaps accurate, and speed up the server.
  • Measure the result with Crawl Stats, server logs and how quickly new products get crawled and indexed.

Table of Contents

Does your store actually have a crawl budget problem?

Start with the thresholds, because they save many teams a quarter of work. Google’s documentation says crawl budget management is primarily for large sites with 1 million or more unique pages that change moderately often, medium or larger sites with 10,000 or more unique pages that change daily, and sites with a large share of URLs that Search Console classifies as “Discovered - currently not indexed.” Google describes those page counts as rough estimates, not hard cut-offs.

Google’s own guidance from 2017 is blunter about small sites. If new pages tend to be crawled the same day they are published, crawl budget is not something to focus on, and a site with fewer than a few thousand URLs will usually be crawled efficiently. A widely used industry guide puts the practical line for most stores lower: under roughly 500 to 1,000 indexable URLs, crawl budget probably is not the main issue.

The catch for ecommerce is that your product count is not your URL count. Filters for size, color, brand and price can each create a new URL. One technical SEO analysis describes a store with 800 products where every category exposes ten combinable filters: Googlebot’s perceived inventory is hundreds of thousands of pages, not 800. That is how mid-size stores cross the “large site” line without anyone noticing.

Does your store actually have a crawl budget problem?

A quick self-check

You are probably in crawl budget territory if any of these are true:

  • New products or restocked items take days or weeks to appear in Google.
  • Search Console shows a growing pile of “Discovered - currently not indexed” URLs.
  • Your crawl stats show Googlebot spending most requests on parameter URLs, internal search or cart paths.
  • You have tens of thousands of products, or a smaller catalog with deep filtering.

If none apply, spend the time on content, internal links and page speed for users instead. If one or more apply, keep reading.

How Google decides how much of your site to crawl

Google’s documentation breaks crawl budget into two parts.

Crawl capacity limit. This is how many parallel connections Googlebot will use and how long it waits between fetches, so it does not overload your server. The limit rises when your site responds consistently and fast, and falls when the site slows down, returns 5xx errors or sends rate-limiting signals such as HTTP 429. Search Console also lets site owners set a ceiling on crawl rate.

Crawl demand. This is how much Google wants to crawl, and it varies with site size, update frequency, page quality and relevance. Google’s 2017 post names three drivers: perceived inventory, popularity and staleness. Without guidance, Googlebot tries to crawl every URL it knows about. If many of those are duplicates or junk, it burns time on them and may get to your important pages later.

Two clarifications help avoid wasted effort:

  • Speed helps, but not as a magic lever. Google says that the faster pages load and render, the more it can crawl, though making the site fast for users matters more than increasing crawl coverage. Rendering time counts as much as request time.
  • A faster server alone will not raise your budget if demand is low. Google notes that improving availability does not necessarily increase crawl budget, and that making low-quality pages faster will not encourage Googlebot to crawl more of them.

The practical read: capacity is mostly an engineering problem you can solve with a stable, quick platform. Demand is a content and architecture problem. Ecommerce stores need both.

What wastes crawl budget on ecommerce sites

Google’s 2017 analysis ranked the low-value URL types that hurt crawling and indexing, in order of significance: faceted navigation and session identifiers, on-site duplicate content, soft error pages, hacked pages, infinite spaces and proxies, and low-quality or spam content. Its current troubleshooting guide adds shopping cart pages to the list. Here is how each shows up in a store.

Faceted navigation and URL parameters

Every combination of size, color, brand, price and sort order can generate its own URL. The result is a huge set of near-duplicate pages that Googlebot can crawl but that rarely deserve to rank. The usual advice is to let Google crawl only one or two useful filters, such as a category plus brand, and keep the rest out. We cover the engineering side in our guide to fixing faceted navigation SEO.

Duplicate paths to the same product

A product reachable at several category paths, with tracking parameters or session IDs, creates duplicates. Consolidate them with one canonical URL pattern and consistent internal links.

Soft 404s and removed products

Pages for discontinued items that return a 200 status with an “out of stock” message look like soft 404s. Google recommends returning a 404 or 410 for permanently removed pages and eliminating soft 404s.

Redirect chains

Migrations and seasonal collection changes leave chains of redirects. Each hop is another request. Point links and sitemaps straight to the final URL.

Internal search, cart and account paths

Internal search results can produce unlimited URLs, and cart or checkout paths have no search value. These are prime candidates for robots.txt rules.

Hacked or injected pages

Compromised stores can sprout thousands of spam URLs that soak up crawling and damage trust. Treating managed website security as part of crawl hygiene keeps injected URLs from piling up unnoticed.

Thin and orphaned pages

Empty category pages, near-empty tag pages and orphan URLs that Google only finds through old sitemaps or backlinks add noise. Fix thin pages or remove them, and link the ones that matter. Our piece on internal linking strategy shows how to find the gaps that matter.

How to see where Googlebot is spending its time

You need evidence before you change robots.txt on a live store. Use three sources.

The Crawl Stats report

Search Console’s Crawl Stats report shows how many requests Google made, when, what the server returned and any availability issues. It is aimed at advanced users and is not needed for sites with fewer than a thousand pages. Look at the breakdown by response code and file type, and check whether requests are rising while your real catalog is not.

Google’s troubleshooting guidance also suggests checking whether Googlebot requests exceeded the red limit line for availability, and using the URL Inspection tool to test for Hostload exceeded warnings.

Server logs

Crawl Stats is a sample and summary. Server logs record every request with its URL, timestamp, user agent and response code, which makes them the complete record of what Googlebot actually fetched. Collect a few weeks of logs, verify that the hits are genuine Googlebot (reverse DNS or Google’s published IP ranges), then group requests by site section. You will see quickly what share goes to product pages versus filters, search and parameters.

A quick ratio from Crawl Stats

A simple urgency check: divide the number of pages on your site by the average pages Google crawls per day. A result well above 10 suggests Googlebot is nowhere near covering your site in a reasonable window. Treat it as a heuristic, not a Google rule. It comes from Yoast’s crawl budget guide, and is best read next to your logs.

Pair it with a site crawl

Crawl your own site with a desktop crawler and compare the list of URLs it finds with what logs show Googlebot requesting. URLs crawled by Google but not in your intended architecture are waste. URLs in your architecture that Google never requests are gaps.

The fixes, in the order we would run them

Order matters. Reducing what exists is more durable than telling Google what to ignore.

  1. Shrink the URL inventory at the source. Decide which filter combinations deserve a page (category plus brand, for example) and stop generating links to the rest. Where possible, apply filters without creating crawlable links.
  2. Consolidate duplicates. Choose one canonical URL pattern per product and category, and make internal links, sitemaps and canonicals agree.
  3. Block low-value URL spaces with robots.txt. Google’s guidance is to block unimportant URLs in robots.txt and not to rely on noindex for this, because a page with noindex still has to be fetched for the tag to be seen. As one analysis of robots directives puts it, noindex still consumes crawl budget. Do not block anything that must be indexed, and do not block pages whose canonical or noindex signals you rely on Google reading.
  4. Return the right status codes. Use 404 or 410 for products that are gone for good, and fix soft 404s. Redirect only where there is a true replacement.
  5. Keep sitemaps clean and current. List only canonical, indexable, 200-status URLs, and update the last-modified date only when content truly changes.
  6. Remove redirect chains. Update internal links to the final destination.
  7. Improve server speed and stability. A consistent, fast response raises the crawl capacity limit. Our website performance guide covers where stores usually lose time.
  8. Flatten the architecture. Important products buried many clicks deep are crawled less. See our notes on site architecture and crawl paths, and on product taxonomy design for how categories should be structured in the first place.

Which control for which problem

For faceted URLs, the controls differ in strength. Robots.txt disallow stops the crawl entirely. Canonicals and nofollow are suggestions and work less reliably over the long term. The old Search Console URL Parameters tool was removed in April 2022, so parameter handling now has to live in your own site and robots rules.

Problem Best first control Watch out for
Filter and sort parameters Fewer crawlable links, then robots.txt Blocking filters you want indexed
Duplicate product paths One canonical pattern, aligned links Canonicals that conflict with sitemaps
Discontinued products 404 or 410 Soft 404s returning 200
Internal search, cart robots.txt Blocking pages that should rank
Redirect chains Update links to final URL Chains left from old migrations
Slow server responses Hosting, caching, rendering fixes Heavy JavaScript rendering

If your store renders products with heavy JavaScript, rendering time counts toward crawl cost too. Our JavaScript SEO audit walks through how to check it.

Perspective

We rarely see a crawl budget problem that is only a crawl budget problem. It is usually a platform decision made years ago: a filter system that was built for shoppers and never for crawlers, a taxonomy that grew by accident, a migration that left redirect chains behind.

That is why we do not treat it as a one-off SEO ticket. The fix touches templates, robots rules, sitemaps, caching and internal linking, and each of those lives in a different team’s backlog. When strategy and engineering sit apart, the recommendation gets lost between the audit and the sprint.

Our model is to keep the senior people who diagnose the problem involved in building the fix. We decide which facets deserve to exist, then implement the rules on the store and measure the effect in logs. For Shopify and BigCommerce stores, that means working inside the platform’s real constraints instead of handing over a generic checklist.

The goal is not a smaller site. It is a site where Google’s attention goes to the pages that earn revenue.

Service

How to tell whether the changes worked

Do not judge success by total crawl volume. A healthy outcome can even be fewer requests, spread across better URLs. Track these instead:

  • Share of Googlebot hits on valuable pages. Take verified crawler requests in your logs and see what percentage land on product and category pages versus parameters, search and redirects. This should rise.
  • Discovery time for new products. Measure the gap between publishing a product and Googlebot’s first request, then its appearance in the index. This should shrink.
  • Crawler error rate. The share of 4xx and 5xx responses to Googlebot should fall.
  • Discovered or crawled but not indexed counts. In Search Console’s page indexing report, the number of important URLs sitting in these buckets should decline over the following weeks.
  • Crawl response time. In Crawl Stats, average response time should stay stable or improve, which supports a higher capacity limit.

Give changes time. Google recrawls on its own schedule, so compare logs before and after over several weeks, and set up a recurring log export so crawl behavior becomes a monitored metric instead of a quarterly investigation.

When this is worth a dedicated engagement

If you have found a crawl problem but the fixes sit across theme code, platform settings, hosting and merchandising, it helps to have one team that owns the whole path from diagnosis to deployment. That is what we do for scaling commerce and multi-location brands. Our growth platforms work covers Shopify and BigCommerce engineering, technical SEO and answer-engine visibility, and engagements run on monthly capacity: Core capacity is $3,500 to $5,999 per month, Growth capacity is $6,000 to $9,999 per month, and Scale capacity starts from $10,000 per month. See the pricing page for what each level includes.

Quicktoimpress

If you want to see how the work fits your stack first, browse our capabilities or the enterprise commerce page.

FAQ

Is crawl budget a ranking factor?

No. Crawl budget controls how much of your site Google fetches, not how pages rank. It matters because a page that is not crawled cannot be indexed or updated in results. The effect is on freshness and coverage, especially for new or changed products.

How many pages does an ecommerce site need before crawl budget matters?

Google points to sites with 1 million or more unique pages, or 10,000 or more pages that change daily. Those are rough estimates. A smaller store with deep faceted filtering can still generate far more crawlable URLs than products, so count URLs, not SKUs.

Should I use noindex or robots.txt for filter pages?

To save crawl budget, use robots.txt, because Google has to fetch a page to see a noindex tag. Noindex is the right tool when a page must be crawled but kept out of the index. Never block a URL in robots.txt if you need Google to read a canonical or noindex on it.

Does page speed increase crawl budget?

Partly. A fast, stable server raises the crawl capacity limit, and Google says faster pages let it crawl more. But speeding up low-quality pages will not make Google want them. Fix demand with better architecture and fewer junk URLs, and fix capacity with a fast platform.

How do I find out what Googlebot is crawling on my store?

Open the Crawl Stats report in Search Console for a summary, then analyze raw server logs for the full record. Verify that requests come from real Googlebot, group them by site section, and compare them against your intended list of indexable URLs.

Sources