Double Descent

Modern deep learning defies classical bias-variance wisdom. After the interpolation threshold, test error descends again — larger overparameterized models generalize better.

0.15
40
0.00
Classical regime (parameters p < samples n): More complexity → lower bias, higher variance → classic U-shape.
Interpolation threshold (p ≈ n): The model just barely fits the data → maximum variance, peak test error.
Modern regime (p > n): Overparameterized models use the minimum-norm interpolating solution → implicit regularization → test error falls again.
Regularization (λ > 0) smooths the peak, revealing the monotone descent favored by gradient descent in practice.