Relevance Scoring: Building a Quantitative Framework for Content Quality

Subjectively evaluating content relevance is unreliable at scale. You need a scoring system that converts qualitative relevance dimensions into measurable,...

Dilshad Akhtar
Dilshad Akhtar
Published: 5 August 2026
5 min read
TL;DRAI summary
  • Subjectively evaluating content relevance is unreliable at scale.

Subjectively evaluating content relevance is unreliable at scale. You need a scoring system that converts qualitative relevance dimensions into measurable, trackable metrics. This post presents a relevance scoring framework that combines entity coverage, topical depth, information gain, and...

Relevance Scoring: Building a Quantitative Framework for Content Quality

Subjectively evaluating content relevance is unreliable at scale. You need a scoring system that converts qualitative relevance dimensions into measurable, trackable metrics. This post presents a relevance scoring framework that combines entity coverage, topical depth, information gain, and resonance into a single composite score. You can implement this framework with standard NLP tools and SERP APIs.

Why Score Relevance?

A numerical relevance score serves three purposes. It provides a consistent standard across writers and editors. It enables A/B testing of content changes against a baseline score. And it creates a data-informed feedback loop: score a page, optimize it, measure the score improvement, and correlate the improvement with ranking changes.

A 2025 study by STAT Search Analytics analyzed 500 pages that were scored using a relevance rubric and then published. Pages scoring in the top quartile of the rubric ranked 62% higher on average than bottom-quartile pages after 90 days, even when controlling for domain authority and backlink count (STAT, 2025). The scoring system itself became a leading indicator of ranking performance.

The Component Scores

The composite relevance score is the weighted sum of four component scores.

Entity Coverage Score (ECS). Count the entities on your page. Compare against the entity union of the top-3 ranking pages. ECS is the ratio of your entity set to the competitor entity union, expressed as a percentage. A page covering 80% of the entity space scores higher than a page covering 40%. Weight in composite: 25%.

Topical Depth Score (TDS). Count the distinct subtopics your page covers that directly relate to the primary query. Divide by the optimal subtopic count for your query type (5 to 8 for most informational queries). Cap the denominator at 8. A page covering 6 of 8 optimal subtopics scores 75%. Weight in composite: 30%.

Information Gain Score (IGS). Count the number of claims, data points, or frameworks on your page that are absent from all top-10 ranking pages. Divide by the total claims on your page. IGS is the uniqueness ratio discussed in post 1489. A page with 10 claims, 4 of which are unique, scores 40%. Weight in composite: 25%.

Resonance Score (RS). Calculate the Jaccard similarity between your page's top-50 vocabulary and the top-50 vocabulary of the aggregate top-3 SERP results. Normalize to a 0 to 100 scale. A Jaccard score of 0.4 becomes a resonance score of 40. Weight in composite: 20%.

Computing the Composite

The composite relevance score (CRS) is:

CRS = (ECS x 0.25) + (TDS x 0.30) + (IGS x 0.25) + (RS x 0.20)

All component scores are on a 0 to 100 scale, so the composite also ranges from 0 to 100. A score above 70 indicates strong relevance. Above 85 indicates exceptional relevance. Below 50 signals a page that needs significant optimization.

Automating the Scoring Pipeline

You can compute these scores programmatically. ECS requires entity extraction (spaCy or GLiNER) and a data source for competitor content. TDS requires a topic model or a precomputed subtopic list per query. IGS requires a text similarity comparison against competitor content, using embeddings or cosine similarity on extracted claims. RS requires a simple vocabulary extraction and Jaccard calculation.

A 2025 implementation guide from Moz describes a relevance scoring system built entirely with Python, using requests for SERP data, spaCy for entity extraction, sentence-transformers for claim similarity, and a lightweight Flask API for serving scores to a content management system (Moz, 2025). The entire pipeline runs on a single server.

Using the Score in Practice

The relevance score is not a ranking guarantee. It is an optimization target. Score each page before and after editing. A score improvement after optimization correlates with ranking improvement in approximately 70% of cases, based on 2025 data from an ongoing study by SEOClarity across 1,200 pages (SEOClarity, 2025).

Track the score at the content brief stage, the draft stage, and the live publishing stage. If the score drops between stages, the change introduced a relevance regression. Catch it before it goes live.

Audit: Scoring System Checklist

  • [ ] Have you defined your entity coverage calculation? Which NER tool will you use?
  • [ ] Is your optimal subtopic count per query type established?
  • [ ] Do you have a method for detecting unique claims (original information gain)?
  • [ ] Is your vocabulary overlap (Jaccard) calculation implemented?
  • [ ] Are the component weights set and documented?
  • [ ] Do you have a process for scoring content at each stage of production?

A relevance scoring system turns content quality from an opinion into a number. That number is the foundation for data-informed SEO decisions. The final post in this series covers implementing relevance engineering workflows in your organization.

References

  1. STAT Search Analytics. (2025). "Relevance Scoring as a Leading Indicator of Ranking Performance." STAT Research.
  2. Moz. (2025). "Building an Automated Relevance Scoring System with Python." Moz Developer Blog.
  3. SEOClarity. (2025). "The Relevance Score Correlation Study: 1,200 Pages Tracked." SEOClarity Research.

Ready to Build Your Dream Website?

Let's discuss your project and create something amazing together.