A new proximity measure published on arXiv offers a mathematically grounded method for identifying whether records drawn from multiple independent data sources describe the same physical object, addressing a longstanding problem in information system design.
Record linkage — the task of matching entries across databases that describe the same real-world entity — underpins everything from healthcare data consolidation to fraud detection and government registries. Existing approaches often require feature values to be transformed into a common format before comparison, introducing additional complexity and potential error. The paper, titled Proximity Measure of Information Object Features for Solving the Problem of Their Identification in Information Systems, proposes an alternative that works directly with raw feature data.
A Measure That Handles Both Numbers and Categories
The core contribution is a hybrid proximity measure that treats quantitative and qualitative features differently but within a single unified framework. For numerical features, the author applies a probabilistic measure — one that accounts for the likelihood that two differing values could represent the same underlying quantity given known or estimated measurement error. For categorical features, a measure of possibility is used instead, drawing on possibility theory rather than classical probability.
This distinction matters in practice. A patient's recorded age in two hospital systems might differ by one year due to data entry conventions; a product's colour description might vary between "navy" and "dark blue." A rigid exact-match system would treat both as mismatches. The proposed measure is designed to tolerate such discrepancies in a principled, mathematically consistent way.
Unlike many known measures, the proposed approach does not require feature value transformation to ensure comparability.
The paper also verifies the measure's mathematical validity by demonstrating compliance with the standard axioms required of any proximity or similarity measure — a step that distinguishes it from ad hoc scoring methods that may produce inconsistent or counterintuitive results in edge cases.
Why Record Linkage Remains a Hard Problem
Data integration at scale is one of the more persistent engineering challenges in modern information systems. When data about the same person, object, or event arrives from multiple independent sources — each with its own format, accuracy level, and terminology — determining which records belong together is far from trivial. The problem is compounded by the absence of a universal unique identifier across many real-world systems.
Most existing solutions fall into one of two camps: deterministic matching, which relies on exact agreement on key fields, and probabilistic matching, which uses statistical models to score the likelihood of a match. The Fellegi-Sunter model, developed in 1969, remains a widely cited foundation for the latter approach. More recent machine learning methods have improved accuracy in some domains but typically require labelled training data — matched record pairs that human annotators have already verified.
The framework proposed in this paper sits closer to the probabilistic and possibility-theoretic tradition, but its authors argue it avoids a common limitation: the need to pre-process or normalise feature values before comparison. This is significant because transformation steps can introduce assumptions that distort the original data or fail when feature distributions differ across sources.
Multiple Variants for Different Use Cases
Beyond the core proximity measure for individual features, the paper proposes several variants for aggregating feature-level proximities into an overall object-level proximity score. This gives practitioners flexibility: a system integrating medical records might weight certain fields — such as date of birth or a national identifier — more heavily than others, while a logistics platform might prioritise location and timestamp data.
The paper does not, according to its abstract, include empirical benchmarks comparing the proposed measure against existing methods on real-world datasets. Whether the theoretical advantages translate into measurable gains in precision and recall on practical record linkage tasks remains to be demonstrated in subsequent work. Researchers evaluating the approach for deployment should note that the current publication presents a theoretical framework rather than an evaluated system.
The work is published as a preprint on arXiv and has not yet undergone formal peer review, meaning its claims, while mathematically argued, have not been independently validated by the wider research community.
What This Means
For data engineers and information system architects dealing with multi-source integration, this framework offers a theoretically rigorous alternative to transformation-dependent matching methods — though empirical validation on real datasets will be needed before adoption in production systems.