The Hidden Cost of Large Weights: Understanding Regularization in Machine Learning
One of the first things I learned in Machine Learning was that reducing loss is important. After all, a lower loss usually means better predictions. So naturally, I assumed that if a model keeps reducing the training loss, it must be getting better. But that assumption isn’t always true. Sometimes a model becomes so good at fitting the training data that it starts memorizing it instead of learning meaningful patterns. As a result, it performs extremely well on […]