Why do we use training, validation and test sets, and how can data leakage occur between them?
Interview preparation resource from Gate Smashers.
The training set fits model parameters, the validation set supports model selection and hyperparameter tuning, and the test set provides a final unbiased estimate after all choices are fixed. Leakage occurs when information unavailable at prediction time, or information from validation/test data, influences training. I prevent it by splitting at the correct entity and time boundary, fitting every preprocessing step only on training data, using pipelines, and keeping the test set untouched until final evaluation.
| Split | Used for | Must not be used for |
|---|---|---|
| Training | Fit model and preprocessing parameters | Final performance claim |
| Validation | Tune hyperparameters, select models and thresholds | Direct parameter fitting |
| Test | One-time final evaluation of the frozen pipeline | Repeated tuning or feature selection |
Purpose of each split
The three splits answer different questions. Training data asks, ‘Can the algorithm learn parameters?’ Validation data asks, ‘Which model and settings should we choose?’ Test data asks, ‘How well should the final frozen procedure generalize to unseen data?’ Repeatedly checking the test result during development turns the test set into another validation set.
Common leakage paths
- Preprocessing leakage: Scaling, imputing, feature selection or dimensionality reduction is fitted before splitting, so statistics from held-out data enter training.
- Target leakage: A feature contains the target directly or contains information created after the outcome occurred.
- Entity leakage: Records for the same user, patient, device or document appear in different splits.
- Temporal leakage: Future observations or future aggregates are used to predict an earlier event.
- Tuning leakage: The test set influences hyperparameters, thresholds, features or model choice.
- Augmentation leakage: Near-duplicate augmented versions of one source example cross split boundaries.
Safe workflow
- Define the real prediction time and the information available at that moment.
- Split raw examples by the correct unit, such as user, group or time.
- Fit preprocessing and feature selection only on each training fold.
- Tune models using validation data or nested cross-validation.
- Freeze the complete pipeline and decision threshold.
- Evaluate once on the untouched test set.
Cross-validation nuance
Random K-fold cross-validation is not automatically correct. Use stratified folds for imbalanced classification, group-aware folds when entities repeat, and time-series validation when the future must be predicted from the past.
Why it matters
Leakage produces an optimistic metric rather than a better model. It is especially dangerous because the evaluation can look excellent while the production system fails immediately.