Fellegi-Sunter Matching
Duplicate detection in civic registries cannot rely on strict string equality. We utilize a highly-optimized implementation of the Fellegi-Sunter probabilistic record linkage theorem to isolate fuzzy duplicates with an accuracy exceeding 99.8%.
The Limitations of Deterministic Matching
Legacy SQL-based deterministic matching (e.g., WHERE FIRST_NAME = 'Robert' AND LAST_NAME = 'Smith') fails in the real world. Clerical typographical errors, OCR scanning artifacts, and the presence of suffixes (Jr., Sr., III) guarantee that direct SQL joins will miss up to 15% of actual duplicate voter registrations.
In 1969, Ivan Fellegi and Alan Sunter published a mathematical framework in the Journal of the American Statistical Association that revolutionized entity resolution. By assigning mathematical weights (log-odds) to the probability of specific field agreements and disagreements, we can compute a total statistical similarity score between any two records.
The Core Theorem
Weight = log₂ ( P(Agreement | Match) / P(Agreement | Non-Match) )
If two records agree on a rare name (e.g., "Zebediah"), the weight is massively positive because the probability of two different people randomly sharing that name is nearly zero. If they agree on a common name (e.g., "John"), the weight is positive, but much smaller.
Marigold's Vector Space Mapping
Marigold's Vector Space Mapping
Running an O(N²) quadratic comparison across a 10-million row state database would require 100 trillion comparisons, melting standard CPU architecture. Marigold optimizes this by projecting the demographic data into a high-dimensional vector space using Locality-Sensitive Hashing (LSH).
- Blocking: We first divide the encrypted dataset into logical "blocks" (e.g., grouping by Soundex of the Last Name and the Year of Birth). This reduces the comparison matrix from O(N²) to a highly manageable subset.
- Vectorization: The remaining candidate pairs are vectorized utilizing Jaro-Winkler distance for strings and Levenshtein distance for numeric IDs.
- Log-Odds Summation: We sum the log-odds weights for every field vector.
| Field | M-Probability (True Match) | U-Probability (Random Chance) |
|---|---|---|
| Date of Birth | 0.98 | 0.0027 (1 in 365) |
| First Name | 0.92 | Dynamic (Based on Census Frequency) |
| Last Name | 0.95 | Dynamic (Based on Census Frequency) |
| Street Number | 0.90 | 0.05 |
Thresholds and the Grey Zone
Thresholds and the Grey Zone
Once the final statistical score is computed for a pair of records, it is evaluated against two critical thresholds: the Upper Bound (μ) and the Lower Bound (λ).
- μScore > Upper Bound
The pair is mathematically guaranteed to be a duplicate. The record ID is flagged and encrypted into the return vault with
severity: HIGH. - λLower Bound < Score < Upper Bound
The pair exists in the clerical grey zone (e.g., father and son residing at the same address). Returned with
severity: MEDIUMfor manual auditor review. - -Score < Lower Bound
The pair is definitively not a match. The algorithm discards the pair silently.
Implementation in the FUZZY_DUPLICATE Module
Implementation in the FUZZY_DUPLICATE Module
When you execute the FUZZY_DUPLICATE anomaly type via the /modules/anomalies/detect endpoint, our algorithmic workers automatically perform the Fellegi-Sunter matrix evaluation against the hashed records provided in your payload. You do not need to configure the M/U probabilities manually; our engines are pre-trained on terabytes of declassified US Census demographic distribution data to ensure hyper-accurate thresholding.