Your model performs very well on training data but poorly on validation and test data. What is happening, how would you diagnose it, and how would you fix it?
Interview preparation resource from Gate Smashers.
This is usually overfitting: the model has learned patterns and noise specific to the training set but does not generalize to unseen data. I would first verify the data split and rule out leakage, then compare training and validation learning curves. Depending on the cause, I would simplify the model, add regularization, collect or augment data, use early stopping, improve cross-validation, or reduce noisy features. I would confirm the fix using an untouched test set rather than the training score.
| Observation | Likely interpretation | Next check or action |
|---|---|---|
| Low training error, high validation error | High variance / overfitting | Learning curves, regularization, simpler model |
| Both training and validation errors are high | High bias / underfitting | More capacity, better features, less regularization |
| Validation is unexpectedly excellent | Possible leakage or duplicate samples | Audit features and split boundaries |
| Offline score is good, production score is poor | Distribution or pipeline mismatch | Compare live features, labels and segments |
A persistently large positive validation-loss gap is a practical signal of overfitting, although leakage and split problems must still be ruled out.
Recognising the pattern
Overfitting is a generalization problem. A very low training error with a much higher validation error indicates that the model has enough capacity to fit the training examples but has learned relationships that do not remain stable on unseen data. A large train–validation gap is evidence of high variance, but it is not enough by itself to identify the root cause.
Diagnose before changing the model
- Check the split: Ensure duplicate users, repeated events, future information and related samples do not appear across training and validation sets.
- Plot learning curves: Compare training and validation loss as the number of training examples or epochs increases. A widening validation gap is a strong overfitting signal.
- Inspect segment performance: Evaluate important classes, user groups, time periods and rare cases separately. A good average metric may hide weak generalization.
- Compare cross-validation folds: Large variation between folds may indicate limited data, unstable features or a non-representative split.
- Establish a baseline: Compare the complex model with a simpler linear model, shallow tree or business-rule baseline.
Corrective actions
- Reduce variance: Use a simpler architecture, shallower trees, fewer parameters or fewer noisy features.
- Regularize: Apply L1/L2 penalties, dropout, pruning, minimum leaf sizes or other model-appropriate constraints.
- Improve the data: Collect more representative examples, fix label noise and use valid augmentation where the domain permits it.
- Stop at the right time: Monitor validation loss and restore the best checkpoint with early stopping.
- Tune correctly: Perform model selection inside cross-validation and keep the final test set untouched.
Important interview distinction
More training data often helps high variance, but it does not repair leakage, a wrong objective or a train–production mismatch. Regularization also creates a trade-off: too much can move the model from high variance to high bias.
Practical example
Suppose a fraud model achieves 99% training accuracy and 84% validation accuracy. Before reducing the neural network, check whether transactions from the same customer were split across datasets or whether a post-transaction feature reveals the label. If the split is clean, compare learning curves, regularize the model and evaluate precision–recall metrics on an untouched time-based test set.