Workflow guide · Machine learning QA

Prove split integrity before trusting an evaluation score

The right split depends on how the model will meet future data. Row separation is only the first check; entities, time, preprocessing state, labels, and post-outcome features can all leak information across the boundary.

Worked sample, not a live evaluation.This guide uses synthetic CSV rows. It does not train a model, estimate accuracy, make a fairness claim, or approve production use.

Create a split-integrity packet Open the paired benchmark Verify receipt + artifact bytes

1. Design the split around deployment reality

Write down the prediction timestamp, unit of prediction, entity/group key, label-availability time, and future population. If the same customer, device, patient, document, or session can appear on both sides, random row splitting may measure memorization instead of generalization.

2. Use a leakage decision tree

Same normalized row on both sides? Quarantine duplicates and trace the data-generation cause.
Same entity or group on both sides? Re-split by group unless deployment explicitly allows that history.
Training data occurs after test prediction time? Rebuild with a chronological cutoff and label-availability delay.
Feature created after the outcome? Exclude it or reconstruct it as it existed at prediction time.
Preprocessor fitted on all data? Fit imputation, encoding, scaling, selection, and representation only on training folds.
Distribution shift detected? Investigate collection and segment changes; pair drift with model and domain evidence.

3. Wrong approaches

Random split by default

Rows may be independent in a file while belonging to the same real-world entity or time sequence.

Drop duplicate IDs only

Copied content, hashed identifiers, session fragments, or derived aggregates can leak without an obvious ID.

Fit transforms before splitting

Global vocabulary, means, selected features, or embeddings can carry test information into training.

Drift means harm

A population statistic is a screening signal. It must be interpreted with performance, segments, and operational consequences.

4. Reproduce the planted leakage

shasum -a 256 \
  app/fixtures/flagship-editorial-20260713/ml/leaky-train.csv \
  app/fixtures/flagship-editorial-20260713/ml/leaky-test.csv

leaky-train.csv982460187e7c0bf8e7cf9192369f7b3371b56bded964f60df80cb0563bdad32b

leaky-test.csv254f6cddf1bdd4b954280973f331e21902499e1e1facda7e086c403b42100040

By inspection, shared-a and shared-b occur in both splits; the shared-a row is copied exactly; and approved_flag equals label for every row. The proper action is to rebuild the split and feature set before model evaluation.

5. Retain an evidence packet

Evidence	Minimum context	Blocker example
Split policy	Unit, group key, cutoff, label delay, random seed or deterministic rule	Policy cannot reproduce membership
Overlap ledger	Exact/near match reason, columns used, quarantine decision	Unresolved cross-split entity overlap
Feature provenance	Creation time, source, transform fit scope, owner	Post-outcome or direct label proxy
Drift review	Metric, threshold rationale, segments, model impact check	Material shift with no analysis or owner
Receipt	Input and artifact hashes, engine identity, assumptions and limits	Reviewed artifacts do not match exported bytes

Method, limits, and privacy

Method: deployment-aligned split design, row/entity/time checks, target-proxy review, train-only preprocessing, drift screening, and retained evidence. Limits: lexical similarity misses semantic duplicates; drift does not prove performance harm; correlation does not prove causality. Privacy: use only public, synthetic, or fully de-identified data—never customer, patient, employee, credential, or financial rows.

Official sources

Correction channel

Send a method, leakage definition, privacy boundary, source, or fixture correction to [email protected] with evidence and a proposed change. FastTool is the accountable publisher; material changes appear in the changelog. Reviewed 2026-07-13.