Neural Tangent Kernel and NN Dynamics (draft)

August 1, 2019 · Xingyu Li · Noise and Generalization · math neural network dynamics review

Discussion on Neural Tangent Kernel The Neural Tangent Kernel is introduced by Arthur Jacot et. al. to study the dynamics of a (S)GD-based learning model, say, the Neural Network. It is expected to enable one to “study the training of ANNs in the functional space $\mathcal{F}$, on which the cost $C$ is convex.” In the following, we will review the derivations of Arthur Jacot et. al. and argue that by definition the Neural Tangent Kernel only captures the first order dynamics of the Neural Networks.

Read More