Pagination and Filters: How Sites Accidentally Generate a Million URLs

Add a few filters to a product listing and you have quietly built a URL generator. Not a page — a generator, capable of emitting more distinct addresses than your catalog has products. Crawlers treat every one of those as a page worth fetching, and the arithmetic is much worse than it feels while you are writing the template.

The arithmetic

5 filters, 4 options each, combinable:
    4 x 4 x 4 x 4 x 4  = 1,024 URLs

...multiplied by 3 sort orders             = 3,072
...multiplied by 20 pages of pagination    = 61,440
...for ONE category

Now put that template on forty categories. You have 2.4 million URLs for a shop with maybe eight hundred products, and each one returns a real 200 response with content on it. Nothing is broken. That is what makes it dangerous.

The consequences are practical rather than punitive. Crawlers spend their time on permutations instead of your actual products, your genuinely new pages get discovered slowly, and Search Console fills with thousands of "Crawled — currently not indexed" URLs that make real problems impossible to spot.

rel=next and rel=prev are not the answer

Clear this out of the way, because it is still in a lot of advice. Google announced in 2019 that it no longer uses rel="next" and rel="prev" for pagination — and noted it had not used them for some years before saying so. Adding them today is harmless and accomplishes nothing at Google. Some other consumers still read them, so it is not wrong, just not a solution.

What Google actually wants from pagination is much more mundane: ordinary links to ordinary URLs.

Pagination that works

Three rules, and they are all about being boring:

1. Each page gets a unique, crawlable URL. ?page=2 or /page/2, either is fine. What is not fine is a listing that only changes state in JavaScript with no URL to link to.
2. Each page self-canonicalizes. Page 3 is not a duplicate of page 1 — it holds different items. Canonicalizing page 3 to page 1 asks Google to drop pages 2 onward and everything reachable only from them.
3. Real anchors. A crawler follows <a href>. It does not click a button that calls a router.

<!-- crawlable -->
<a href="/shoes?page=2" rel="next">Next</a>

<!-- invisible to crawlers -->
<button onclick="loadPage(2)">Next</button>

Infinite scroll is the same problem wearing nicer clothes. If it is your interface, back it with real paginated URLs and update the address bar with the History API as the user scrolls, so every position in the list is a linkable, crawlable page.

Sorting facets from filters

The useful distinction is whether a combination is something a person would search for. Split your parameters into three buckets:

WORTH INDEXING
  /running-shoes?brand=acme
     People search "acme running shoes".
     Real demand, distinct content.

NOT WORTH INDEXING
  /running-shoes?sort=price_desc
  /running-shoes?view=grid
     Same products, different arrangement.
     Nobody searches for this.

ACTIVELY HARMFUL
  /running-shoes?brand=acme&color=red&size=42&sort=price&view=grid
     Combinatorial. Thin. Infinite.

Bucket one gets indexed like any other page. Buckets two and three need to stop being crawled, and you have several tools with quite different trade-offs.

The tools, and when each is right

Controlling crawl of filtered URLs

Tool	Effect	Use when
Disallow in robots.txt	Not crawled at all	Infinite spaces. The heavy tool, and the right one here.
noindex	Crawled, not indexed	A finite set of pages you want kept out of results.
rel="canonical"	Consolidates to a base URL	Sort and view variants of genuinely identical content.
Do not render the link	Never discovered	The most effective option, and the most overlooked.

Note the trap in combining the first two. noindex only works if the page is crawled, so blocking a URL in robots.txt and marking it noindex means the noindex is never read.

For the infinite case, robots.txt patterns are the practical answer:

User-agent: *
# sorting and display state
Disallow: /*?sort=
Disallow: /*&sort=
Disallow: /*?view=
# session noise
Disallow: /*?sessionid=

Sitemap: https://example.com/sitemap.xml

Both ?sort= and &sort= are needed, since the parameter can appear first or later in the query string.

Keep your URLs canonical-friendly

Two habits that cut duplicate variants dramatically:

• Fixed parameter order. ?brand=acme&size=42 and ?size=42&brand=acme are different URLs with identical content. Sort parameters alphabetically when you build links.
• Drop empty parameters. ?brand=&size= should just be the base URL. Filter them out before rendering.

One historical note so you do not go looking for it: Search Console used to have a URL Parameters tool for exactly this, and Google retired it in 2022. The replacement is what is described above — do it in your own markup and robots.txt.

Spotting a trap you already have

Search Console's Crawl Stats report shows what Googlebot is spending its requests on. If parameterized URLs dominate, that is your answer. Your own access logs are even more direct:

# the most-requested query strings from crawler traffic
grep -i 'googlebot' access.log \
  | awk '{print $7}' | grep '?' \
  | sed 's/.*?//' | sort | uniq -c | sort -rn | head -20

If sort=price_desc is in the top five, Googlebot is cataloging your sort orders instead of your products.

The short version

1. Paginated pages: unique URLs, real anchors, self-canonical. Skip rel=next/prev.
2. Decide per parameter whether anyone would search for it.
3. Block infinite parameter spaces in robots.txt; canonicalize finite duplicates.
4. Never block and noindex the same URL.
5. Best of all: do not link to combinations you do not want crawled.