Bayes Classifier and MI Classifier With Noisy Labels

August 8, 2019 · Xingyu Li · Noise and Generalization · review math information theory

Definition and Relation This section is a partial review of Bao-Gang Hu’s paper. Bayes Classifier without rejection and its Decision Rule Bayes Classifier makes decisions based on a (often) subjectively designed risk function $\mathfrak{R}$: $$\mathfrak{R}(y_j | x) = \sum_i \lambda _{ij} P(x | t_i)P(t_i),$$ where $x \in \R^d$ is the input feature; $y_j$ and $t_i$ stand for the predicted class and the true class, respectively; $\lambda _{ij}$ is the cost when a true label $i$ is classified as $j$.

Neural Tangent Kernel and NN Dynamics (draft)

August 1, 2019 · Xingyu Li · Noise and Generalization · math neural network dynamics review

Discussion on Neural Tangent Kernel The Neural Tangent Kernel is introduced by Arthur Jacot et. al. to study the dynamics of a (S)GD-based learning model, say, the Neural Network. It is expected to enable one to “study the training of ANNs in the functional space $\mathcal{F}$, on which the cost $C$ is convex.” In the following, we will review the derivations of Arthur Jacot et. al. and argue that by definition the Neural Tangent Kernel only captures the first order dynamics of the Neural Networks.