Start Here — One Record's Lineage

This three-minute tour follows one synthetic fan across the actual pipeline output. The selected cluster is chosen by SQL from the current generated build and is verified against generator ground truth as exactly one person — names, identifiers, match evidence, warehouse keys, totals, and that verification are not typed into this page. (What a cluster looks like when the matcher gets it wrong is shown further down, on purpose.)

Choose a path: Data engineers can follow the six numbered stages below. Analytics consumers can jump to the BI-ready result.

1. Land and validate source records

The pipeline extracts a Salesforce-shaped API plus ticketing, merchandise, and email files into an S3-compatible landing zone. Pandera contracts normalize valid records and quarantine invalid rows with reasons.

Prefect flow · contracts · quarantine loader

2. Compare identity evidence

Ground-truth check: these 9 source records belong to 1 true identity — verified against ground truth at build time by the evaluation harness, the only component allowed to read it.

match_evidence is derived from the normalized source rows and the persisted cluster method. A deterministic cluster can contain transitive edges, so the table does not invent a pair-level reason when the current model does not retain one. Probabilistic records show the retained score; ambiguous pairs below the auto-merge threshold go to manual review instead of this cluster.

deterministic rules · probabilistic matcher · threshold evidence

3. Resolve the golden identity

The unifier writes the source-to-fan cross-reference and applies deterministic survivorship rules. Stable cluster membership produces a stable fan_id.

golden-record implementation · identity schema

4. Preserve dimension history

Identity attributes use SCD2 history; activity attributes update in place so a new purchase does not create a meaningless identity version.

SCD model · design decision

5. Join a warehouse fact

ticket fact model · referential-integrity check

6. Consume the BI-ready row

Fan 360 mart · data dictionary · pipeline operations · measured linkage quality

Anatomy of a false merge

The matcher is measured, not perfect — and the dominant failure mode deserves the same visibility as the success story. This is the largest cluster the ground truth says is wrong: its records belong to more than one real (generated) person, merged because a household shares one email address.

Shared household emails are the measured top false-positive source (see recall by mess type); the review band exists so ambiguous pairs go to a human queue instead of merging, and household modeling is the roadmap's next evidence-led experiment. Hiding this cluster would contradict the evaluation this project publishes.

Trust boundary

• Every displayed person and transaction is deterministically generated and fictional.
• The featured cluster is verified pure against ground truth at build time; if no verified cluster exists, the site build fails rather than showing an unlabeled one.
• CI rebuilds the source systems, pipeline, warehouse, and site together.
• PostgreSQL and MinIO prove the local interfaces; Redshift and S3 remain a documented, validate-only deployment design.
• The first probabilistic threshold performed worse than deterministic rules. That result remains visible rather than being removed from the story.