Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization

arXiv:2401.15604v4 Announce Type: replace-cross
Abstract: Diffusion models have become a leading paradigm in generative AI, with score estimation via denoising score matching as a central component. While recent theory provides strong statistical guarantees, it typically relies on algorithm-agnostic assumptions and treats empirical risk minimization as if it were solved exactly. In practice, however, score functions are parameterized by highly nonconvex neural networks and trained by gradient descent (GD), and it remains unclear whether such practical procedures admit rigorous guarantees.
We take a first step toward this question by developing a mathematical framework for score estimation with GD-trained neural networks. Our analysis addresses both optimization and generalization. We introduce a parametric formulation that reduces denoising score matching to a regression problem with noisy labels. This setting poses several challenges, including unbounded inputs, vector-valued outputs, and an additional time variable, which prevent a direct application of existing techniques. We show that, with a suitable design, the dynamics of GD-trained networks can be approximated by a sequence of localized kernel regression problems. We also show that prolonged training on noisy labels leads to overfitting, and derive an early-stopping rule adapted to unbounded domains. As a consequence, we establish the first minimax-optimal generalization bounds for GD-trained neural networks in diffusion models. Experiments on the Credit Default dataset further show that our theory-guided training framework achieves performance comparable to heavily tuned heuristic methods for generating high-fidelity financial tabular data.