What is Semantic Similarity in SEO? How to Measure it

How Is Semantic Similarity Calculated?

Text becomes a list of numbers called an embedding. Each embedding is a position in space. Cosine similarity measures the angle between two positions. That is the entire mechanism.

What Is an Embedding?

A model reads a piece of text and converts it into a long list of numbers, often 768 or 1,536. That list is called an embedding, or a vector. Every text gets its own list

.

How Do Embeddings Represent Meaning?

Think of a map. On an ordinary map, two numbers place a shop in Kochi: latitude and longitude. An embedding works the same way with 1,536 directions instead of two.

Texts about ayurveda land in one region of that space. Texts about car insurance land far away. Texts about panchakarma land beside the ayurveda ones.

What Is Cosine Similarity?

Two texts that mean the same thing point in nearly the same direction. The angle between them is small and the score approaches 1. Unrelated texts point elsewhere and the score falls.

Embedding plus cosine similarity equals semantic similarity. Everything else in this article is application.

Worth knowing:  Sentence-BERT, introduced by Nils Reimers and Iryna Gurevych in 2019, made this practical at scale. Finding the closest sentence among 10,000 took the original BERT roughly 65 hours, according to the authors. SBERT does the same job in about five seconds. That speed difference is why embedding-based analysis is available to you inside a crawler today.

Does Google Use Semantic Similarity to Rank Pages?

The 2024 Google Content Warehouse API leak exposed three attributes that are the same maths described above: siteEmbedding, siteFocusScore and siteRadius. Topical authority turned out to have a formula.

Most articles on this topic treat semantic similarity as a technique SEOs borrowed from machine learning. The leaked documentation suggests it is closer to a mirror of what Google already computes.

Three attributes matter here:

Read that back slowly. Google builds a topical identity for your domain, then measures every page against it. A page far from the centre is measurably an outlier, not just an intuition.

This changes what pruning means. Removing or reworking an off-topic post is not housekeeping. It is a direct adjustment to a number in the scoring function.

It also explains why the outlier feature in modern crawlers works the way it does. They average every page embedding to find a centroid, then measure deviation from it, which is the publicly reproducible version of siteRadius.

At What Levels Should You Measure Semantic Similarity?

Word, chunk, page and query. Most SEOs work only at page level, which is why their AI search results stay flat.

The chunk level is where AI search is determined. ChatGPT, Perplexity, Gemini and Google AI Overviews do not retrieve your page. They retrieve a passage from it and build an answer around that passage.

A page can score respectably as a whole and still fail, because the one paragraph that should answer the question is thin. Page-level averages hide that failure completely.

What Can You Use Semantic Similarity For in SEO?

Cannibalization detection, topical focus repair, internal linking, relevance diagnosis, redirect mapping and chunk-level GEO work.

How Do You Find Keyword Cannibalization With Semantic Similarity?

Keyword-based cannibalization checks find two pages targeting the same phrase. Semantic similarity finds two pages that mean the same thing while using different vocabulary.

Example:  A page titled “Ayurvedic treatment for back pain” and another titled “Kizhi therapy for spinal problems” can score 0.94 against each other. Google reads one topic. The site published two pages. Neither ranks properly, and no keyword tool flags the conflict.

How Do You Fix a Site That Has Lost Topical Focus?

Average your page embeddings to find your site centroid. Any page sitting far from it is pulling your topical identity sideways.

An ayurveda site with three Kerala tourism posts and one article about GST filing has a measurable focus problem. 

How Do You Choose Internal Links Using Semantic Similarity?

Rank every page by similarity to every other page, then link the closest pairs. That produces a semantic network instead of a link web assembled by memory and guesswork.

Why Is My Page Not Ranking for Its Target Keyword?

Embed the target query. Embed your page. Embed the top five results. If you score 0.62 against the query while they score 0.85, the gap is meaning, not word count.

How Do You Map Redirects Using Embeddings?

For a migration with 4,000 URLs, match each old URL to its closest new URL by embedding distance. Two weeks of manual mapping becomes an afternoon of review.

How Do You Optimise Content Chunks for AI Search?

Take the twenty questions your audience actually asks. Embed each one. Embed every paragraph on your page. Score each question against each paragraph.

Any question with no paragraph above roughly 0.75 is unanswered on that page, whatever you believe you wrote. Fill those gaps inside the body copy, where the question belongs in context, rather than in a block bolted to the bottom of the page.

What Is a Good Semantic Similarity Score?

These bands are working defaults, not constants. Thresholds are model-dependent and must be calibrated on your own data before you act on them.

What Should You Check Before Acting on a Similarity Score?

Similarity is not intent. “Buy ayurvedic oil online” and “how ayurvedic oil is made” can score 0.88 while serving completely different intents. Confirm on the SERP before merging anything.

High similarity to a competitor is not a win. Scoring 0.97 against the top result means you added no information gain and gave the engine no reason to prefer you.

There is no universal threshold. The same pair of texts scores differently under different embedding models, so the bands above have to be validated against a sample you review by hand.

The caveat almost nobody mentions:  Researchers at Netflix and Cornell published a 2024 paper, Is Cosine-Similarity of Embeddings Really About Similarity?, showing that cosine similarity can produce arbitrary values depending on how the underlying model was regularised. Their conclusion was not that the method is useless, but that it should not be treated as ground truth. In SEO terms: use the score to find candidates, then use your judgement and the SERP to make the decision.

Sitebulb tested alternative distance algorithms against the same embedding data and found cosine similarity remained the practical choice for content analysis. The method holds up. The blind trust does not.

Which Tools Check Semantic Similarity for SEO?

Screaming Frog v22 and above is the working tool for whole-site analysis. Google Sheets covers small manual jobs. Colab covers chunk-level work. SERP overlap is the free proxy.

How Do You Check Semantic Similarity for Free?

Search your target query, then a second query you suspect is the same topic. If the results overlap by more than roughly 60 percent, Google treats them as one topic.

This costs nothing and reflects Google’s live judgement rather than a generic model’s opinion. Use it as a cross-check on every merge decision.

How Do You Check Semantic Similarity in Screaming Frog?

From version 22.0, the SEO Spider analyses semantic similarity across a crawl using LLM embeddings, going beyond text matching to surface duplicate, similar and off-topic pages. You need a paid licence and an API key from an AI provider such as OpenAI, Gemini, or a local Ollama model.

Setup runs in this order:

1. Config, then API Access, then AI. Add your key and confirm the connection under Account Information.
2. In Prompt Configuration, use Add from Library and select the pre-built embeddings prompt, which uses the SEMANTIC_SIMILARITY task type.
3. Config, then Spider, then Extraction. Enable Store HTML and Store Rendered HTML so page text is available for embedding.
4. Config, then Content, then Embeddings. Tick Enable Semantic Similarity, and Low Relevance if you want outlier detection.
5. Crawl the site.
6. Run Crawl Analysis afterwards. Skipping this leaves every similarity column empty.

Results live in the Content tab under the Semantically Similar filter. The Semantic Similarity Score column shows similarity to the page named in Closest Semantically Similar Address.

Setting that matters:  The default threshold is 0.95 and can be lowered to 0.5. At 0.95 you catch only near-identical pages. Drop it to 0.85 for most audits, because that band is where real cannibalization lives.

The content area used for embeddings is configurable under Config, then Content, then Area. Navigation and footer elements are excluded by default, which matters: messy input produces meaningless embeddings.

Cost is minor. Embedding a 500-page site runs to a few rupees of API credit.

How Do You Calculate Cosine Similarity in Google Sheets?

For twenty or thirty texts, no crawler is needed. Pull embeddings from an embeddings API, paste each vector into a row, then compute the score with one formula.

=SUMPRODUCT(A2:ZZ2, A3:ZZ3) / (SQRT(SUMPRODUCT(A2:ZZ2, A2:ZZ2)) * SQRT(SUMPRODUCT(A3:ZZ3, A3:ZZ3)))

That formula is the whole calculation. Running it manually once removes the mystery, which matters if you plan to teach this to anyone.

How Do You Run Semantic Similarity Analysis in Python?

Use Colab for chunk-level analysis, question-to-paragraph matching, or clustering thousands of URLs. It runs in the browser with nothing installed locally.

Read your URLs from a CSV, then either call an embeddings API such as text-embedding-3-small, or run sentence-transformers locally with a model like all-MiniLM-L6-v2 at zero API cost. Compute the matrix with cosine_similarity from scikit-learn, then export it. Add KMeans or HDBSCAN for automatic clusters and UMAP to plot them.

Around thirty lines of code in total.

Which SEO Tools Do Not Measure Semantic Similarity?

How Do You Run a Semantic Similarity Audit Step by Step?

One audit for cannibalization, one diagnosis for a page that will not rank. Both take under two hours.

How Do You Audit a Whole Site for Cannibalization?

7. Crawl with embeddings enabled and the threshold at 0.85, then run Crawl Analysis.
8. Export the Semantically Similar filter from the Content tab.
9. For every pair above 0.90, ask three questions: same intent, similar SERPs, and does either page hold an independent role in the topical map?
10. Same intent with no independent role: merge the weaker into the stronger and 301 it.
11. Different intent: keep both, rewrite the overlapping sections, and repair internal links so each page points at its own query.

How Do You Diagnose a Page That Will Not Rank?

12. Embed the target query, your page, and the top five ranking pages.
13. A meaningfully lower score against the query means the page is missing meaning, not keywords.
14. Embed fifteen real audience questions and score each against each of your paragraphs.
15. Any question with no paragraph above roughly 0.75 marks a hole in the main content.
16. Write the missing answers into the body, in the section where each question naturally arises.

What Are the Most Common Semantic Similarity Mistakes in SEO?

Most wasted effort comes from six errors, and half of them are threshold errors.

What Should You Learn After Semantic Similarity?

This covers measurement and decisions. It does not cover embedding model selection, vector databases, or RAG pipeline construction.

Three areas sit outside this scope and deserve separate study. Choosing between embedding models on cost, dimension count and multilingual support is one. Storing vectors at scale in a database built for them is another. Building a retrieval pipeline that mimics how an AI engine reads your site is the third.

If you are starting today, the sequence that produces results fastest is:

• Run the SERP overlap check on your five most confusing keyword pairs. No tooling required.
• Crawl your own site with embeddings at a 0.85 threshold and read the outlier list before the duplicate list.
• Take your best-performing page and run the question-to-paragraph check on it at chunk level.

The third one changes how people write. Once you have seen a well-written page score 0.58 against a question its author was certain it answered, page-level thinking stops feeling sufficient.

Prasad Karthik is an SEO trainer and consultant based in Thiruvananthapuram, and the creator of Organic Growth Mastery, an advanced AI-first content-led SEO course.

Sources referenced

• Screaming Frog SEO Spider documentation, semantic similarity and embeddings configuration (v22.0 onwards).
• Google Content Warehouse API leak, May 2024: siteEmbedding, siteFocusScore and siteRadius attributes, as analysed by Search Engine Land and others.
• Steck, Ekanadham and Kallus (Netflix and Cornell), Is Cosine-Similarity of Embeddings Really About Similarity?, arXiv 2403.05440, March 2024.
• Reimers and Gurevych, Sentence-BERT, 2019, for the sentence-pair speed comparison.
• Sitebulb, comparative testing of distance algorithms against embedding data for content analysis.