Every company of any size has the same quiet problem. One customer exists five times. The CRM knows them as Robert J. Okafor, billing has Bob Okafor, the support desk typed Kestral instead of Kestrel, and a 2019 acquisition brought in a sixth record nobody has touched since. Revenue reporting counts them separately. The fraud model treats them as strangers. The AI assistant you just deployed cheerfully answers questions about all six as though they were unrelated.
Entity resolution is the work of deciding which records describe the same real thing, and being able to defend that decision afterwards. It sounds like a data cleaning chore. In practice it is the foundation that analytics, risk scoring and retrieval all quietly depend on, which is why getting it slightly wrong is so expensive and so hard to notice.
Start with deterministic matching, and know where it stops
Deterministic matching applies exact rules. If two records share a national tax identifier, or a verified email address, or an account number, they are the same entity. No scoring, no judgement.
This should always be the first pass, for two reasons. It is cheap, and it is explainable. When an auditor asks why two records merged, "identical verified email" is an answer that ends the conversation. The technique goes back to the foundational work on record linkage that the US Census Bureau record linkage research group has been publishing for decades, and the logic has not changed much since.
The limit arrives quickly. Deterministic rules only fire on fields that are both present and clean. In most enterprise data the high-confidence identifiers are missing on a third of rows or more, and when they are present they are often the records you had no trouble with anyway. Exact matching finds the duplicates you could have found by sorting a spreadsheet. It misses the ones costing you money.
Probabilistic matching: scoring, not guessing
The second pass compares fields that are similar rather than identical, and assigns a score. String similarity measures do the heavy lifting here. Jaro-Winkler handles transpositions and truncations well, which makes it suited to names. Levenshtein distance counts edits and suits short codes. Token-based measures work better on addresses, where word order moves around but the words themselves survive.
The important design decision is not which measure you pick. It is what you do with the score. A well-built resolver has three bands, not two:
- Above the upper threshold, merge automatically. Typically 0.90 or higher on a calibrated scale.
- Below the lower threshold, leave the records alone and log nothing.
- Between them, route to a human review queue with the evidence attached.
Teams that skip the middle band always regret it. Set one threshold and you are choosing between merging strangers and missing duplicates, with no way to buy accuracy at the margin. The middle band is where you convert analyst time into precision, and where you gather the labelled examples that let you tune the model later.
Open tooling has matured considerably here. Splink from the UK Ministry of Justice implements the Fellegi-Sunter model at scale and is genuinely production grade. The Python Record Linkage Toolkit covers the classic comparison and classification steps in a familiar API, and dedupe takes an active learning approach that gets useful results from a few hundred labelled pairs.
Where graph structure beats pairwise comparison
Pairwise matching asks one question over and over: are these two records the same? That framing hides the most valuable signal in the data, which is that duplicates are rarely isolated pairs. They form clusters, and the clusters have shape.
Consider three records where no single pair clears your threshold. Record A and record B share an address but the names differ. Record B and record C share a phone number but the addresses differ. Record A and record C look unrelated on their own. Pairwise scoring rejects all three comparisons. A graph sees one connected component held together by two shared attributes, and the component is far more convincing than any of its edges.
This is where modelling your records as a graph pays for itself. Records become nodes, shared attributes become nodes too, and the edges are the observations. Then you run weakly connected components to find candidate clusters, and node similarity or a weighted score to decide which clusters are tight enough to collapse. Both are standard operations in the Neo4j Graph Data Science library, and they run over hundreds of millions of relationships without leaving the database.
The graph framing also gives you something pairwise matching cannot: a natural place to stop. A cluster that keeps growing as you add edges is usually a sign that you have linked on an attribute that is not identifying. Shared corporate address, shared shipping locker, shared prepaid phone. Watching cluster size distribution is the fastest way to catch a bad matching rule before it reaches production.
Blocking, and why the naive version never ships
Comparing every record against every other record is quadratic. At one million records that is five hundred billion comparisons, which is why the first honest version of every resolution pipeline is too slow to run. Blocking fixes it by only comparing records that share a coarse key: the same postcode, the same name phonetic code, the same year of birth.
Blocking is also the step most likely to silently destroy your recall. Every record pair that does not share a block is a pair you will never compare, and therefore never merge. The mitigation is to run several independent blocking passes with different keys and take the union. Postcode catches the movers who kept their name, name phonetics catch the marriages, and email domain catches the corporate accounts. Any one of them alone leaves a predictable hole.
Decide early whether resolution runs in batch or at write time. Batch is simpler and fine for analytics. Write-time resolution, where a new record is matched against the existing graph before it lands, is what you need for onboarding checks and fraud scoring, and it constrains your design considerably: the whole match has to complete in the time a human will wait for a form to submit.
Golden records, and keeping the evidence
Merging is not deleting. The output of a good resolution pipeline is a golden record that represents the entity, plus a full record of which source rows contributed to it, which rule or score caused the merge, and when. This is what master data management practice calls lineage, and it is the difference between a system your compliance team will sign off on and one they will not.
Keep the sources. Every merge you make will eventually be wrong for some record, and when that happens you need to be able to split the entity back apart without reloading history. Systems that overwrite the source rows cannot do this, and the recovery is always a project.
Survivorship rules decide which value wins when sources disagree. Most useful default: prefer the most recently verified value from the most authoritative source, and record the alternatives rather than discarding them. For regulated work, particularly anything touching know your customer obligations under FinCEN rules, the ability to show the full set of observed values is not optional.
Measuring it honestly
Resolution quality is a precision and recall problem, and you cannot improve what you do not measure. Build a labelled evaluation set early, even a small one. Two hundred manually adjudicated pairs, drawn from across the score distribution rather than only from the easy ends, will tell you more than any amount of eyeballing.
Watch both error types separately, because they cost different amounts. A false merge combines two real customers and is visible to them, which makes it the expensive one in consumer businesses. A missed merge leaves duplicates in place and is expensive in analytics and risk, where it quietly understates exposure. The right threshold depends on which error you can better afford, and that is a business decision rather than a technical one.
Re-measure after every change to the upstream data. Resolution quality decays. A new source system, a changed address format, a partner who starts sending names in all capitals, any of these will move your score distribution without moving your code.
What this looks like when it works
A production resolver runs in layers. Deterministic rules first, because they are free and unarguable. Blocking next, to reduce the comparison space from quadratic to something tractable, usually on a coarse key like postcode or name soundex. Probabilistic scoring within blocks. Then graph clustering over the surviving links, a review queue for the middle band, and a golden record with lineage at the end.
Built that way, the work is measurable and it moves real numbers. H-E-B improved identity resolution by 50% after a resolution engagement. The same foundations underpin the fraud detection work described here, because a fraud ring is structurally the same problem: records that look unrelated until you model what they share.
If you are staring at a duplicate problem and are not sure whether it is a matching problem or a modelling problem, that distinction is usually the whole engagement. It is also the kind of thing worth thirty minutes on a call before anyone writes code.