Original lab · Machine learning QA

Will a split-integrity review catch copied rows, repeated entities, and a target proxy?

Two tiny synthetic datasets make the failure modes inspectable by eye. The clean pair separates entity IDs and time windows. The leaky pair repeats two entities, copies one complete row, and includes a feature equal to the label.

Worked sample, not a live model result.The observations are direct counts from twelve synthetic CSV rows. They do not report current workbench output, model accuracy, causality, production drift, or risk to real people.

Run ML Dataset Leakage & Drift Lab Read the workflow guide Verify receipt + artifact bytes

Benchmark design

Clean split

Four train and four test rows have disjoint entity IDs. All train timestamps are in January 2026 and all test timestamps are in February 2026.

Leaky split

Three train and three test rows share two entity IDs. One row is copied exactly. The synthetic feature approved_flag equals label in all six rows.

Method

Confirm train and test schemas, label, entity/group key, time field, and excluded post-outcome fields.
Hash normalized rows across the agreed shared feature columns to identify exact overlap.
Compare entity/group keys across splits even when row content or timestamps differ.
Check time ordering and fit all learned preprocessing only on training data.
Screen features for direct or near-direct association with the label, then investigate provenance rather than declaring causality.
Measure numeric/categorical drift as a review signal and pair it with model-performance and domain evidence.

Fixture hashes

shasum -a 256 app/fixtures/flagship-editorial-20260713/ml/*.csv

good-train.csvb4bbb9748ede7001723e5d444076fbb4ba4bfcd74867842157085cab505aff8c
good-test.csv90b1eeb00c4583661f103bf5088156dda57305e51e2753b7ac3e97a354d5ed0d
leaky-train.csv982460187e7c0bf8e7cf9192369f7b3371b56bded964f60df80cb0563bdad32b
leaky-test.csv254f6cddf1bdd4b954280973f331e21902499e1e1facda7e086c403b42100040

Deterministic fixture observations

Case	Rows train/test	Shared entity IDs	Exact copied rows	Direct target proxy	Decision
Clean split	4 / 4	0	0	None declared	Continue review
Leaky split	3 / 3	2	1	`approved_flag = label` in 6/6 rows	Block and rebuild split

“Continue review” is intentionally not “production ready.” Eight rows cannot establish representative drift, model utility, fairness, or generalization. The clean fixture only lacks the three planted defects.

Limits and privacy

Limits

Exact equality and lexical near-duplicate checks can miss semantic duplicates.
Drift statistics do not prove performance harm, and association does not establish causality.
Small fixture counts are for inspection, not statistical inference.

Privacy boundary

Use only public, synthetic, or fully de-identified rows. Do not paste customer, patient, employee, credential, or financial records. Browser-local parsing does not make sensitive data authorized or anonymous.

Official sources

Corrections

Report a row count, leakage definition, source, privacy boundary, or hash mismatch to [email protected] with evidence and a proposed change. FastTool is the accountable publisher; material corrections appear in the changelog. Reviewed 2026-07-13.