高斯判别分析

全称是Gaussian Discriminant Analysis (GDA)。大家不要被名字所误导，这是一种概率生成模型。

原理

对联合概率进行建模，我们假设$y\sim Bernoulli(\Phi )$，且$x|y=1\sim N({\mu }_1,\Sigma )$，$x|y=0\sim N({\mu }_0,\Sigma )$

输入

训练集数据$D=(x_1,y_1)...(x_M,y_M)$，$x_i\in \mathcal{X}\subseteq R^n$，$y_i\in \mathcal{Y}\subseteq R^K$，二分类 y i ∈ { − 1 , + 1 } y_i \in \{-1, +1\} yi​∈{ −1,+1}

$X_i,{\mu }_i,{\Sigma }_i$ 分别表示第i类样例的集合、均值向量、协方差矩阵。

输出

各分类的概率。如果是二分类就是{+1, -1}的概率。

损失函数

令$\theta =({\mu }_0,{\mu }_1,\sigma ,\Phi )$。计算似然
$$\begin{gathered} L(\theta )=\sum _{i=1}^{M}log[P(x_i|y_i)P(y_i)] \\ =\sum _{i=1}^{M}logP(x_i|y_i)+logP(y_i) \\ =\sum _{i=1}^{M}log[N({\mu }_1|\Sigma {)}^{y_i}N({\mu }_0|\Sigma {)}^{1-y_i}]+log[{\Phi }^{y_i}(1-\Phi {)}^{1-y_i}] \\ =\sum _{i=1}^{M}log[N({\mu }_1|\Sigma {)}^{y_i}]+log[N({\mu }_0|\Sigma {)}^{1-y_i}]+log[{\Phi }^{y_i}(1-\Phi {)}^{1-y_i}] \end{gathered}$$
对$\Phi$进行求偏导，
$$\begin{gathered} \frac{\partial L(\theta )}{\partial \Phi }=\sum _{i=1}^M\frac{y_i}{\Phi }+\frac{1-y_i}{1-\Phi }=0 \\ \sum _{i=1}^M\Phi (1-y_i)-(1-\Phi )y_i=0 \\ \sum _{i=1}^M[y_i-\Phi ]=0 \\ \Phi =\frac{1}{M}\sum _{i=1}^M{y}_i \end{gathered}$$
对${\mu }_1$进行求偏导，

$$\begin{gathered} \sum _{i=1}^{M}log[N({\mu }_1|\Sigma {)}^{y_i}]=\sum _{i=1}^M{y}_{i}log[\frac{exp(-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1))}{(2\pi {)}^{p/2}(\Sigma {)}^{1/2}}]=0 \\ {\mu }_1^{\ast }=argma{x}_{{\mu }_1}\sum _{i=1}^M{y}_i(-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1)) \\ =-\frac{1}{2}\sum _{i=1}^M(x_i^T{\Sigma }^{-1}-{\mu }_1^T{\Sigma }^{-1})(x_i-{\mu }_1) \\ \frac{\partial L(\theta )}{\partial {\mu }_1}=\sum _{i=1}^M{y}_i(-{\Sigma }^{-1}{x}_i+{\Sigma }^{-1}{\mu }_1^T)=0 \\ {\mu }_1^{\ast }=\frac{{\sum }_{i=1}^M{y}_i{x}_i}{{\sum }_{i=1}^M{y}_i} \end{gathered}$$
类似的，我们可以推导出
$${\mu }_0^{\ast }=\frac{{\sum }_{i=1}^M(1-y_i)x_i}{{\sum }_{i=1}^M(1-y_i)}$$
用$S$记录矩阵的方差，我们先对$L(\theta )$与之相关的部分进行化简，
$$\begin{gathered} L(\theta )=\sum _{i=1}^{M}log[N({\mu }_1|\Sigma {)}^{y_i}]+log[N({\mu }_0|\Sigma {)}^{1-y_i}]+log[{\Phi }^{y_i}(1-\Phi {)}^{1-y_i}] \\ =\sum _{k=1}log[N({\mu }_1|\Sigma )]+\sum _{k=0}log[N({\mu }_0|\Sigma )]+\sum _{i=1}^{M}log[{\Phi }^{y_i}(1-\Phi {)}^{1-y_i}] \\ -------------------------------------- \\ \sum _{i=1}^{M}logN(\mu ,\Sigma )=\sum _{i=1}^{M}log[\frac{exp(-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1))}{(2\pi {)}^{p/2}|\Sigma {|}^{1/2}}] \\ =log[\frac{1}{(2\pi {)}^{p/2}}]+log[|\Sigma {|}^{-\frac{1}{2}}]-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1) \\ =\sum _{i=1}^{M}C-\frac{1}{2}log|\Sigma |-\frac{1}{2}(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1) \\ =C-\frac{1}{2}\sum _{i=1}^{M}log|\Sigma |-\frac{1}{2}\sum _{i=1}^M(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1) \\ =C-\frac{M}{2}log|\Sigma |-\frac{1}{2}\sum _{i=1}^{M}tr[(x_i-{\mu }_1{)}^T{\Sigma }^{-1}(x_i-{\mu }_1)] \\ =C-\frac{M}{2}log|\Sigma |-\frac{1}{2}\sum _{i=1}^{M}tr[(x_i-{\mu }_1{)}^T(x_i-{\mu }_1){\Sigma }^{-1}] \\ =C-\frac{M}{2}log|\Sigma |-\frac{1}{2}tr[\sum _{i=1}^M(x_i-{\mu }_1{)}^T(x_i-{\mu }_1){\Sigma }^{-1}] \\ =C-\frac{M}{2}log|\Sigma |-\frac{M}{2}tr[S\ast {\Sigma }^{-1}] \end{gathered}$$

用$M_k$表示各类数据的个数，对$\Sigma$进行求偏导，
$$\begin{gathered} L(\theta )=-\frac{M_1}{2}[log|\Sigma |-tr(S_1\ast {\Sigma }^{-1})]-\frac{M_0}{2}[log|\Sigma |-tr(S_0\ast {\Sigma }^{-1})] \\ =-\frac{M}{2}log|\Sigma |-\frac{M_1}{2}tr(S_1\ast {\Sigma }^{-1})-\frac{M_0}{2}tr(S_0\ast {\Sigma }^{-1}) \\ ------------------------------- \\ \frac{\partial L(\theta )}{\partial \Sigma }=-\frac{1}{2}[(M{\Sigma }^{-1})-M_0{S}_0^T{\Sigma }^{-2}-M_1{S}_1^T{\Sigma }^{-2}]=0 \\ M\Sigma =M_1{S}_0+M_1{S}_1 \\ {\Sigma }^{\ast }=\frac{M_1{S}_0+M_1{S}_1}{M} \end{gathered}$$

适用场景

无。

优点

1. 鲁棒性较好

缺点

1. 需要数据服从分布（具有严格假设）
2. 参数较多，计算相对比较复杂

Reference

• 白板推导系列，shuhuai007