Explain gradient descent, its main variants, and the practical issues you check when training does not converge.
Interview preparation resource from Gate Smashers.
Gradient descent minimizes a differentiable loss by repeatedly moving parameters in the direction opposite to the gradient. Batch gradient descent uses the full dataset, stochastic gradient descent uses one example, and mini-batch gradient descent uses a small batch and is the common practical choice. If training does not converge, I inspect the learning rate, feature scaling, gradients, loss implementation, initialization, batch size and optimization schedule.
The parameter vector θ moves opposite to the gradient of objective J. The learning rate η controls the step size.
| Variant | Examples per update | Main trade-off |
|---|---|---|
| Batch | Entire training set | Stable direction but expensive updates |
| Stochastic | One example | Very noisy but cheap individual updates |
| Mini-batch | Small batch | Efficient hardware use with manageable noise |
How the update works
At the current parameter vector, the gradient indicates the local direction of greatest increase in the objective. Moving in the negative-gradient direction should reduce the loss for a sufficiently small step. Training repeats forward computation, loss calculation, backpropagation and parameter updates.
Main variants
- Batch gradient descent: Computes each update from the full training set. The direction is stable but each step can be expensive.
- Stochastic gradient descent: Updates from one example. It is noisy and inexpensive per step, but hardware utilization may be poor.
- Mini-batch gradient descent: Uses a batch of examples. It balances gradient noise, throughput and memory use and is standard for neural networks.
Learning-rate behaviour
A learning rate that is too large can cause oscillation, divergence or NaN values. A rate that is too small creates slow progress or apparent plateaus. Schedules such as warm-up, decay or cosine annealing alter the step size during training. Adaptive optimizers such as Adam maintain running estimates of gradient moments, but they do not remove the need to tune the learning rate or validate generalization.
Debugging non-convergence
- Verify the implementation: Test the loss, labels, masking and a tiny batch that the model should be able to overfit.
- Inspect numerical health: Track loss, gradient norms, parameter norms, NaNs and exploding or vanishing activations.
- Normalize inputs: Features on radically different scales can make optimization difficult.
- Tune the step: Sweep learning rates before changing many other settings.
- Check capacity and objective: Optimization cannot fix a model that cannot represent the pattern or a loss that does not match the task.
Important interview point
Gradient descent finds a useful minimum; it does not guarantee the global minimum for general non-convex objectives. In modern deep learning, optimization quality and generalization quality are related but not identical.