What is the Curse of Dimensionality, and how does it affect ML models?
The Curse of Dimensionality is the set of problems that arise when the number of features (dimensions) becomes very large relative to the number of samples.
Sparsity — data becomes increasingly sparse in high-dimensional space; the volume grows exponentially.
Distance becomes meaningless — in high dimensions, the difference between the nearest and farthest neighbor shrinks. Distance-based algorithms (KNN, K-Means) degrade.
Overfitting — more dimensions = more parameters = easier to overfit, especially with small datasets.
Computational cost — training and inference time grow steeply with dimensions.
Mitigation: dimensionality reduction (PCA, UMAP, t-SNE), feature selection, regularization, more data.
Mention the geometric intuition: in high dimensions, distances become nearly equal between all points, so distance-based methods (KNN, K-Means) lose their meaning. This insight separates a textbook answer from a deeper one.