Duplicate work records and count integrity
One request can appear as an email, a ticket, and a project task. Counting every record as a separate work unit inflates arrivals and completions. Collapsing similar records too aggressively creates the opposite error by merging distinct requests.
The Census Bureau's record-linkage standard requires defined link criteria, tested procedures, and evaluation of automated and clerical results. Its statistical scope differs from a business workflow, but those controls fit a count-integrity audit.
Define the unit before matching records
State what the metric counts: a customer request, approved deliverable, incident, appointment, or another observable unit. Record how follow-ups, reopened items, subtasks, forwarded messages, and channel transfers relate to that unit. Without this rule, duplicate detection becomes a search for similar rows rather than a test of metric identity.
Preserve every source identifier and parent-child link. Exact shared identifiers can support deterministic links, but reused subject lines or customer names do not prove identity. Create separate outcomes for confirmed same unit, confirmed different unit, candidate match, and unresolved.
Protect the source record. Do not delete or overwrite a record merely because the analysis links it to another. The Census editing standard calls for retaining unedited values, which allows later review of how a count changed.
Test exact and candidate matches separately
Run exact rules first, such as a stable external request ID paired with a source-system rule. Then apply documented candidate rules using fields known to be relevant, such as normalized contact, receipt window, subject, reference number, or attachment identity. Freeze thresholds before reviewing metric effects.
Draw a review sample from matched, unmatched, and borderline pairs. Two reviewers can independently apply the unit definition to a subset, then reconcile disagreement. Publish reviewed error counts and unresolved coverage instead of presenting the match process as perfectly accurate.
The Census linkage research overview describes record linkage as a method for identifying duplicates within and across files. It also treats linkage as a data-quality process, not as proof that every proposed match is correct.
Recompute counts under defensible rules
Show raw record count, confirmed unit count, candidate-linked count, and unresolved count. Recalculate arrival, completion, and reopen measures under a strict rule and a documented broader rule. The difference is sensitivity to linkage assumptions, not a new observed truth.
Check whether candidate duplicates concentrate by source, period, work type, or integration change. A sudden rise after a system migration may point to an ingestion rule, but the pattern alone does not establish the cause.
An operations coordinator can maintain the linkage ledger, review queue, and count reconciliation. Data owners approve identity rules, access, retention, and privacy controls. Our operations support services can maintain this evidence, while the Research index contains related data-quality studies.
Sources and limits
- Census record-linkage standard, for link criteria, testing, clerical review, and error monitoring.
- Census linkage research overview, for duplicate identification within and across files.
- Census linkage technical documentation, for governed ingest, linkage, and documented use.
- Census linkage infrastructure, for metadata, methods, and prior-linkage results.
- Census linkage policy overview, for necessity, privacy, and review considerations.
- Census editing and imputation standard, for duplicate checks and retained source values.
- Census administrative records standard, for source quality, coverage, and processing.
- Census data management standard, for reproducible processing and retained documentation.
- GAO data reliability guide, for accuracy, completeness, and applicability.
- Census record-unduplication recommendations, for validation and unduplication research needs.
These sources support a record-linkage audit, not permission to combine personal data or a universal match threshold. Candidate links can remain uncertain. This article is operational research guidance, not privacy, legal, audit, or data-governance advice.