Explain bias versus variance. How do you identify which problem a model has, and how do you choose the right remedy?
Interview preparation resource from Gate Smashers.
Bias is error caused by assumptions that are too simple, while variance is sensitivity to the particular training sample. High-bias models underfit and usually perform poorly on both training and validation data. High-variance models fit training data well but perform much worse on validation data. I diagnose them using training and validation errors, learning curves and cross-validation stability, then adjust capacity, features, regularization and data accordingly.
Expected prediction error can be viewed as squared bias, variance and irreducible noise. The exact decomposition shown applies to squared-error regression.
Core idea
Bias and variance describe two different ways a model can fail to generalize. Bias comes from an overly restricted model or representation that cannot capture the real relationship. Variance comes from a model that reacts too strongly to the details or noise of its training sample.
How to diagnose them
- High bias: Training performance is already weak, and validation performance is similarly weak. Adding more data alone usually gives limited improvement because the model cannot fit the underlying pattern.
- High variance: Training performance is strong, but validation performance is significantly worse. Performance may also vary widely across cross-validation folds.
- Both: A real system can have high bias in one segment and high variance in another. Inspect slices rather than relying only on a global metric.
How to reduce high bias
- Use a more expressive model or richer feature representation.
- Reduce excessive regularization.
- Train longer when optimization has not converged.
- Add meaningful interactions, nonlinear features or domain knowledge.
- Verify that the target and loss function represent the actual task.
How to reduce high variance
- Add more representative training data.
- Simplify the model or constrain its depth and capacity.
- Increase appropriate regularization.
- Remove unstable or noisy features.
- Use bagging, early stopping or robust cross-validation.
Important interview point
The goal is not to minimize bias or variance independently. The goal is to minimize expected error on unseen data. Increasing model capacity may reduce bias but increase variance, while strong regularization may reduce variance but increase bias.