Detecting Duplicates With Machine Learning in 2026: From String Metrics to Embeddings and LLMs

The more data companies gather on their customers, products, and code, the more duplicated information accumulates, confusing the teams who depend on it. You don't go through millions of records manually. So deduplication is now a machine learning problem. This article looks at the current state of duplicate detection in ML systems where the old techniques still work, and what embeddings and large language models have brought to the table.

How Do ML Systems Detect Duplicates?

It is possible for a person to look at two pictures or two strings and know if they are the same. Teaching a machine to do the same raises difficult questions. What is 'like'? There are degrees of resemblance. Is “Jon Smith, 12 Oak St.” the same customer as “Jonathan Smith, 12 Oak Street?”

In practice, modern deduplication pipelines work like a funnel that begins with cheap, fast methods and introduces intelligence (and cost) only where necessary:

• Normalize block. Clean the data (case, punctuation, abbreviations, etc.) and group records into blocks where they share something in common (e.g., postal code or the first letters of a name). Only records within the same block are compared.
• Score candidate pairs using string metrics. Fast deterministic similarity measures catch typos, transpositions, and formatting differences.
• Embeddings for meaning. Dense vectors and approximate nearest neighbor search can discover pairs that are worded differently but mean the same.
• Push borderline cases up. If the earlier stages cannot be confidently settled, an LLM or a human reviewer resolves the pairs.

Each stage decreases the number of pairs that the next, more expensive stage has to consider. This makes the approach affordable at scale.

String Metrics Still Matter

String metrics are the first line of defense. They are fast, interpretable, and do not need a model. Some common options are:

• Hamming distance: The number of positions at which two strings of equal length differ, or the number of substitutions needed to change one string into the other.
• Levenshtein distance: Generalizes Hamming to allow insertions and deletions as well as substitutions, so it works on strings of different lengths.
• Jaro-Winkler similarity: Measures the number of matched characters and transpositions, with more weight for a common prefix. It works particularly well with short strings, such as personal names.
• Sørensen–Dice coefficient: A measure of similarity based on the number of common bigrams (adjacent letter pairs).
• Jaccard similarity and TF-IDF cosine similarity: Comparing sets of tokens or weighted vectors of words; the latter is more suitable for longer text.
• MinHash with locality-sensitive hashing (LSH): a cheap way to estimate set similarity, so you can find near-duplicate documents without comparing them all.
• Learnable distance: understands that different domains have different weights for edit operations and learns those weights from labeled data.

In a modern pipeline, these scores are typically features that go into a classifier, not the final answer. For example, a model can learn that a mismatch in a phone number is much more important than a mismatch in a middle initial.

Vector Search and Embeddings

String metrics break down when two texts say the same thing in different words. Take two questions from a Q&A site.

1. How can I get cheap hotel rooms?
2. What are the best ways to get cheap hotel deals?

They have very few words in common, and thus have a rather high edit distance rating. Sentence-embedding models, by contrast, map each text into a vector space where similar texts are near each other. Then finding duplicates becomes a nearest-neighbor search:

from sentence_transformers import SentenceTransformer
import faiss

questions = [
    "How do I book cheap hotel rooms?",
    "What are the best ways to find cheap hotel deals?",
    "How do I renew my passport?",
]

model = SentenceTransformer("all-MiniLM-L6-v2")
vectors = model.encode(questions, normalize_embeddings=True)

# Inner product on normalized vectors equals cosine similarity
index = faiss.IndexFlatIP(vectors.shape[1])
index.add(vectors)

scores, neighbors = index.search(vectors, k=2)
for i, (row_scores, row_ids) in enumerate(zip(scores, neighbors)):
    for score, j in zip(row_scores, row_ids):
        if j != i and score > 0.7:  # tune this threshold on labeled data
            print(f"Possible duplicate: {questions[i]!r} ~ {questions[j]!r} ({score:.2f})")

The flat index above is fine for thousands of records. For millions, switch to an approximate index such as HNSW or IVF, which trades a little recall for large speedups. Before reaching for approximate search, check whether simple blocking already shrinks your comparison space enough.

The similarity threshold is not universal. Tune it on a labeled sample of your own data, because the right cutoff for support tickets will differ from the right cutoff for customer names.

LLMs for the Tough Cases

Embeddings provide you with candidates, not certainty. Two product listings might differ only in a model number that matters, but otherwise be nearly the same vector. That’s where big language models are earning their place now, as a pair resolver that looks at two candidate records and decides whether they refer to the same real-world thing.

Open-source tooling has started to package this pattern. One example is cocoindex, which uses FAISS embedding similarity to find near-duplicate entity names, and then queries an LLM-based resolver to confirm matches and choose the better canonical name. Academic work is moving in the same direction. A SIGMOD 2026 paper on in-context, clustering-based entity resolution with LLMs includes ways to reduce hallucinations and API calls.

There are two caveats:

• Cost and delay. Send LLM calls only for the pairs the cheaper stages couldn't resolve, as LLM calls are much more expensive than string comparison. That’s why the funnel is there.
• Matches that have been hallucinated. An LLM can confidently speak of a wrong merge. Keep confidence scores, log reasoning, and have a human in the loop for high-stakes merges.

Real World Examples

Questions on Q&A Sites: Duplicates

Sites such as Quora and Reddit are constantly bombarded with the same questions over and over again because users don't know someone else has already asked them. The most famous benchmark here is the Quora Question Pairs dataset, which came out in 2017 with about 404,000 labeled training pairs [check exact counts; the original paper reported 404,290 training pairs and 2,345,795 test pairs]. The benchmark is hard because duplicates are meaning-based, not word-based; capitalization, abbreviations, and context all matter. The above hotel example is a good first test for any approach. Embedding models do a much better job on this task than edit distance.

Marketplace Duplicate Listings

On classified-ad and marketplace sites, sellers will often re-post or slightly edit a listing if it's not performing. The same funnel applies: block by category and location, score titles and descriptions with string metrics, use embeddings to catch rewording, and flag the clusters for removal or consolidation. Another signal could be added for image similarity when listings have photos.

Copy Code

The biggest example of how much duplication builds up at scale is GitHub. Its Octoverse 2025 report says the platform now has 630 million total repositories, with 121 million added in 2025 alone, and 395 million of them are public or open source. It has more than 180 million developers. At that size, duplicated code is a must.

Code duplication can be exact copies or clones that behave the same, but are written differently. Research on the topic (including a widely cited 2017 study of code duplication on GitHub, DéjàVu) found that a large share of JavaScript files were copies [verify the study’s details and the 93% figure before citing it, and make clear that this was academic research rather than something GitHub itself runs].

The newer story is AI-assisted coding. GitClear research of 211 million changed lines of code in 2025 found an eightfold increase in 2024 in code blocks that duplicate adjacent code with five or more lines. It also found for the first time that copy-pasted lines outnumbered moved (refactored) lines. GitClear sells code-review analytics, and its data shows correlation with AI adoption, not proof of cause, so treat it as vendor research. But it hints that code deduplication will grow in importance as AI tools make it easier to generate new code than to consolidate it.

Duplicate Records in CRM Software

Traditional CRM de-duplication relies on matching rules, and rules can’t cover all fuzzy cases. A learning-based approach, however, identifies fuzzy duplicates by combining multiple string metrics and learning from human feedback: the system shows you a pair of records, you mark them as duplicate or distinct, and it adjusts its matching accordingly. You can learn more about this in my previous article. 

The second big advantage is scalability. Comparing every record with every other record is quadratic. Now suppose you have 50,000 records and add 5,000 new records. Which means 50,000 x 5,000 = 250,000,000 comparisons to compare each new record with each existing record. That is almost 7 hours, 10,000 comparisons per second:

250 000 000 / 10000 / 3600 ≈ 6.94 hours

If you compare every record with every other record in the combined 55,000 records, you get roughly 1.5 billion pairs or about 42 hours at the same speed. Blocking does this by grouping records with common characteristics and only comparing within each group, which reduces the amount of work to be done dramatically. Vector search does something similar, returning only the nearest neighbors of each record.

AI Pipelines Duplicates

Deduplication has also become a core step in the artificial intelligence systems themselves:

• Training data: Removing near-duplicate documents, often using MinHash and LSH, so models don't over-weight repeated text.
• RAG corpora: Collapsing repeated passages so retrieval doesn’t fill the context window with copies of the same content.
• Knowledge graphs: Marrying differently named entities like “Mr. Edwards,” “Brad Edwards,” and “Detective Edwards,” so relationships don’t break across duplicate nodes. Some tools now leverage a combination of deterministic rules, semantic similarity, LLM reasoning, and human review to do this.
• Pitfalls to plan for: False merges are worse than missed duplicates. Merging two different customers can corrupt the records and break downstream processes. Choose the more costly error and set the limits accordingly

Just measure it. Evaluate on a labeled sample of pairs with precision, recall, and F1. Re-check as your data changes.

Keep Humans in the Loop

Low-confidence pairs are passed to reviewers, and their decisions are fed back into the system.

A few considerations remain important:

• Data privacy: Records about people often contain sensitive information. Confirm what you send to hosted embedding or LLM services, and apply your organization's data-handling policies.
• Provenance: Keep original records and a merge log so that a bad merge can be undone.

Machine learning will play an increasingly important role in deduplication, but it doesn't eliminate the need for human judgment.

Any CRM, database, or codebase has a real price for duplicate data, and hand-written filters cannot handle the variety of ways duplicates show up. What the last few years have taught us in practice is not that one technique has replaced another. It’s that the best systems layer them: string metrics and blocking for speed, embeddings for meaning, and LLMs or people for the hardest calls. Start with the cheapest stage that works on your data, evaluate it, and only add the next stage if the results justify the cost.