Compare a decision tree, random forest and gradient boosting. When would you choose each one?
Interview preparation resource from Gate Smashers.
A decision tree is interpretable and fast but can overfit. A random forest trains many trees independently on bootstrapped data and random feature subsets, then averages them to reduce variance. Gradient boosting builds trees sequentially so each stage corrects the current model’s errors, often achieving stronger tabular-data accuracy but requiring careful tuning. I choose based on interpretability, accuracy, latency, data size and operational constraints, then validate rather than assuming one method always wins.
| Property | Decision Tree | Random Forest | Gradient Boosting |
|---|---|---|---|
| Interpretability | High when shallow | Lower | Lower |
| Overfitting risk | High when deep | Reduced by averaging | Controlled through shrinkage and regularization |
| Training | Single model | Independent trees | Sequential trees |
| Typical use | Explainable baseline | Robust nonlinear baseline | High-quality tabular model |
Decision tree
A decision tree recursively splits the feature space to reduce impurity or prediction error. Its rule path is easy to inspect, and it handles nonlinear interactions without feature scaling. A deep tree, however, can change substantially with small data changes and overfit the training set.
Random forest
A random forest applies bagging. Each tree is trained on a bootstrap sample and considers a random subset of features at each split. Trees are trained largely independently, and their predictions are averaged or voted. This decorrelates errors and reduces variance compared with one deep tree.
Gradient boosting
Boosting adds weak learners sequentially. Each new tree is fitted to improve the current ensemble according to the loss gradient. Libraries such as XGBoost, LightGBM and CatBoost include regularization and systems optimizations. Boosting often performs extremely well on structured data but is more sensitive to depth, learning rate, number of trees, leakage and noisy labels.
Choosing in practice
- Choose a small decision tree when a transparent rule set or very low inference complexity is more important than maximum accuracy.
- Choose a random forest for a robust nonlinear baseline with limited tuning and parallel training.
- Choose gradient boosting when tabular predictive performance is the priority and careful validation and tuning are available.
- Consider calibration, model size, feature availability, missing values, latency and explanation requirements before deployment.
Important interview point
Random forest mainly reduces variance through independent trees and averaging. Gradient boosting mainly reduces residual error through sequential correction. Neither automatically solves class imbalance, leakage or distribution shift.