Semantic Search Algorithms: From TF-IDF to Dense Retrieval

Search engines use a two-stage retrieval pipeline: candidate generation and re-ranking. The algorithms at each stage have evolved dramatically over three decades, and understanding this evolution is essential for aligning content strategy with how search engines find and rank pages.

BM25 (Best Matching 25) remains the most widely deployed sparse retrieval algorithm. It extends TF-IDF with document length normalization and term frequency saturation, preventing keyword stuffing from producing unbounded score increases (Robertson & Zaragoza, 2009).

The parameter k_1 controls term frequency saturation (typically 1.2), and b controls length normalization (typically 0.75). BM25 is still used in first-stage retrieval because it is computationally efficient and interpretable.

For SEO, BM25 signals that term frequency still matters at the retrieval stage, but with diminishing returns. Writing a term 10 times instead of 5 times produces minimal improvement. Writing it 100 times instead of 10 produces almost none.

Modern search engines use gradient-boosted decision trees for re-ranking. Google's RankBrain operates alongside traditional LTR features. LambdaMART, an influential LTR algorithm, uses pairwise loss to optimize for NDCG (Burges, 2010).

LTR models use hundreds of features including BM25 scores, PageRank variants, document freshness, user click-through rates, and entity salience. The model learns feature weights from human-rated search quality evaluation data. Google releases thousands of these ratings through Search Quality Raters guidelines.

The implication for SEO is that no single signal dominates. LTR models distribute weight across many features. Over-optimizing for one signal while neglecting others produces worse results than balanced optimization.

The transition from sparse to dense retrieval marks the most significant algorithmic shift since PageRank. Dense retrieval uses dual encoders: a query encoder and a document encoder, both typically Transformer-based, that map queries and documents into the same dense vector space.

The Dual Encoder architecture, formalized in DPR (Dense Passage Retrieval), uses a two-tower model where the query and document are encoded independently (Karpukhin et al., 2020). The relevance score is the dot product of the query and document vectors. This architecture allows pre-computation of document vectors, making first-stage retrieval feasible at web scale.

ColBERT introduces a compromise between dual encoders and cross-encoders. Instead of encoding into a single vector, ColBERT retains token-level embeddings and computes maximum similarity between each query token and all document tokens (Khattab & Zaharia, 2020). This captures fine-grained term matches efficiently. ColBERTv2 is used in production systems for balancing speed and accuracy.

For the re-ranking stage, cross-encoders jointly encode the query and document as a single input. Cross-encoders are more accurate than dual encoders because each token attends to tokens in both the query and document. However, they are too expensive for first-stage retrieval because the full input must be processed for every query-document pair.

Dense retrieval rewards semantic similarity over lexical overlap. If your content does not share any exact words with a query but expresses the same concept, a dense retriever can still match it. This is why keyword stuffing is increasingly ineffective.

Bi-encoder document representations are fixed at indexing time. When a query is issued, its vector is compared against all pre-computed document vectors. This means your content must be semantically self-contained. Context that depends on other pages on your site is not captured in the document vector.

Late interaction models reward token-level precision. ColBERT-type systems still check matching between individual query terms and document terms. Exact matches at the token level still provide signal, especially for rare or domain-specific terms.

• [ ] Verify content uses semantically rich language that would cluster near target queries in embedding space
• [ ] Ensure each page is semantically self-contained (no assumptions about reader having seen other pages)
• [ ] Check that rare/domain-specific terms appear explicitly to provide token-level matching signals
• [ ] Test content-query embedding similarity using sentence-transformers (all-MiniLM-L6-v2 or similar)
• [ ] Review BM25 scores for target queries to ensure baseline lexical relevance is adequate

The retrieval algorithms underlying search engines are more sophisticated today than at any point in history. Understanding sparse versus dense retrieval, bi-encoders versus cross-encoders, and late interaction versus full attention directly informs how to structure content for maximum discoverability.