Most duplicate content advice is about canonicals. The real problem is earlier than that.
Canonical tags influence which URL Google prefers. They don’t stop Google from making a similarity judgment before the canonical even gets considered. There’s a step in the indexing pipeline that most SEO guides skip: near-duplicate detection runs as part of document analysis, and if your page scores too close to an existing document — on your own site or anywhere else on the web — it may simply not earn a unique index slot.
This isn’t a penalty. It’s a resource allocation decision. Google’s index isn’t infinite. Storing documents that add no informational value costs compute, storage, and serving latency. So the system filters aggressively before pages ever compete for rankings.
What the similarity engine is actually measuring
Google uses a family of fingerprinting and hashing techniques — SimHash is the most documented externally — to produce a compact representation of a document’s content. Pages with fingerprints that fall within a threshold distance from an existing indexed document get flagged as near-duplicates. The threshold isn’t public. The features being fingerprinted go beyond raw text: structure, heading hierarchy, term distribution across sections, and entity coverage all feed the representation.
This comparison doesn’t just happen within your domain. Google maintains a web-scale index of document fingerprints. If your page’s coverage of a topic structurally resembles a high-authority page on the same topic — even without copying a single sentence — the similarity score can still push it toward the duplicate threshold.
That last part is what trips up most content teams. You can write everything from scratch, pass any plagiarism checker, and still produce a document that the fingerprinting layer treats as informationally redundant to something Wikipedia or a high-DA domain already has indexed.
Where this actually shows up in practice
A few failure modes come up repeatedly in audits:
- Definition-first content structures. Articles that open with a definition, then a history section, then a list of benefits, then a conclusion — this template is so common that pages on adjacent topics end up with nearly identical structural fingerprints. The topics can differ and the indexer still flags high similarity based on how the document is organized.
- Programmatic pages with light variation. Location pages, product variant pages, or service pages built from templates where the unique content is a single paragraph or a handful of swapped terms. The non-variable scaffold is heavy; the unique content is thin. The fingerprint skews toward the template, not the unique content.
- Republished data with commentary. Take a published dataset or report and wrap 300 words of analysis around it, and the underlying content mass still dominates the fingerprint. The original contribution needs to be the majority of the document, not the framing around it.
The part canonicals don’t solve
Canonical tags tell Google which URL to credit. They don’t suppress duplicate detection. A page can have a canonical pointing to itself, pass a crawl, and still fail to earn an independent index entry because the similarity engine flagged it before the canonical instruction was evaluated.
Google’s documentation frames canonicalization as a signal for URL selection, not content uniqueness. Those are different problems. If your content fails the uniqueness threshold, the canonical is irrelevant — there’s no URL to select because no new document was admitted.
This is also why audits that only check for canonical errors miss the actual indexing problem. You can have perfect canonical hygiene and still have a substantial share of pages sitting outside the index because they didn’t clear the similarity bar.
How to audit for this specifically
Standard indexing audits look at Coverage reports in Search Console and flag non-indexed pages. Necessary, but not sufficient. Cross-reference these signals:
- Pages that Googlebot has crawled (confirmed via log files or the URL Inspection API) but haven’t been indexed — these are the prime candidates for similarity rejection, not crawl or index lag.
- Crawled-not-indexed pages that pass all the obvious checks (no noindex, no robots.txt block, no redirect chains, self-referencing canonical) — similarity rejection is the most common unexplained cause in this bucket.
- Clusters of pages built from similar templates where indexing rate drops below what you’d expect given PageRank distribution — this pattern often means the fingerprinting is collapsing structurally similar pages together.
One underused diagnostic: take two pages you suspect are being collapsed, strip them to plain text, and run a manual diff. If the structural skeleton is more than 60–70% shared after removing the unique body content, you have a fingerprint problem, not a canonical problem.
What actually changes the fingerprint enough to matter
The fix is not adding more words. Length is a poor proxy for informational uniqueness, and similarity scoring doesn’t reward verbosity. What moves the fingerprint:
- Original data or evidence. Primary research, internal metrics, proprietary observations — anything that can’t be reconstructed from existing indexed sources. This genuinely shifts the document’s fingerprint toward something the index hasn’t seen.
- Structural differentiation. Not just unique sentences, but a different organizational logic for the information. If every competitor opens with a definition and closes with a FAQ, building your document around failure modes or decision criteria changes the fingerprint at a structural level, not just a surface one.
- Entity density in non-obvious configurations. Covering a topic through combinations of entities and relationships that don’t already exist in the indexed corpus. This is harder to engineer deliberately, but it’s what genuinely expert content tends to produce on its own.
One honest caveat
Google has never published the exact similarity threshold, the full feature set going into the fingerprint, or the precise algorithm version running at any given time. Everything here is inferred from patents, engineering papers, observed behavior in real audits, and what Google has said publicly about near-duplicate detection. Some threshold specifics could be wrong or domain-dependent.
What’s not uncertain: the mechanism exists, it runs before ranking, and it explains a category of indexing failures that canonical fixes don’t touch. If you have a cluster of crawled-not-indexed pages that look technically clean, run the structural diff before you touch the canonical setup. Nine times out of ten, you’re solving the wrong problem.