高斯判别分析

  高斯判别分析（Gaussian discriminative analysis ）是一个较为直观的模型，属于生成模型的一种，采用一种软分类的思路，所谓软分类就是我们对一个样本决定它的类别时使用概率模型来决定，而不是直接由函数映射到某一类上。生成模型通过求解联合概率来求解 $P(y|x)$。它假设
$y \sim Bernoulli(\phi) \\ x|y=1 \sim N(\mu_1,\Sigma) \\ x|y=0 \sim N(\mu_2,\Sigma)$
  则有
$\begin{aligned} &P(y)=\phi^y(1-\phi)^{1-y} \\ &P(x|y)=N(\mu_1,\Sigma)^y·N(\mu_2,\Sigma)^{1-y} \end{aligned}$
  模型的参数为
$\theta=(\mu_1,\mu_2,\Sigma,\phi)$
  对于生成模型，我们要求解的目标函数是
$\hat y=\arg \max_{y \in \{0,1\}}p(y|x)=\arg \max_yp(y)p(x|y)$
  定义似然函数，则
$\begin{aligned} \hat \theta &=\arg \max_\theta l(\theta) \\ &=\arg \max_\theta \log \prod_{i=1}^Np(x_i,y_i) \\ &=\arg \max_\theta \log \prod_{i=1}^Np(y_i)p(x_i|y_i) \\ &=\arg \max_\theta \sum_{i=1}^N(\log N(\mu_1,\Sigma)^{y_i} \\&+\log N(\mu_2,\Sigma)^{1-y_i}+\log \phi^{y_i}(1-\phi)^{1-y_i})\\ \end{aligned}$

• 求 $\phi$：
  $\begin{aligned} &\frac{\partial l(\theta)}{\partial \phi}=\sum_{i=1}^Ny_i\frac{1}{ \phi}-(1-y_i)\frac{1}{1-\phi} = 0 \\ &\iff \sum_{i=1}^Ny_i(1-\phi)-(1-y_i)\phi=0 \\ &\iff \sum_{i=1}^N(y_i-\phi)=0 \\ &\iff \sum_{i=1}^Ny_i-N\phi=0 \\ &\iff \hat \phi =\frac{1}{N}\sum_{i=1}^Ny_i =\frac{N_1}{N}\\ \end{aligned}$
• 求 $\mu_1,\mu_2$：
    两个的求解过程其实是相同的，所以我们直接求解 $\mu_1$，由于我们只对 $\mu_1$求解，所以原式可以化简为
  $\begin{aligned} &\sum_{i=1}^Ny_i\log \frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\exp(-\frac{1}{2}(x_i-\mu_1)^T\Sigma^{-1}(x_i-\mu_1)) \\ &=\sum_{i=1}^Ny_i\log \frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\exp(-\frac{1}{2}(x_i^T\Sigma^{-1}-\mu_1^T\Sigma^{-1})(x_i-\mu_1))\\ &=\sum_{i=1}^Ny_i\log \frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\exp(-\frac{1}{2}(x_i^T\Sigma^{-1}x_i-2\mu_1^T\Sigma^{-1}x_i+\mu_1^T\Sigma^{-1}\mu_1)) \end{aligned}$
    对上式求导并令导数为0，有
  $\begin{aligned} &-\frac{1}{2}\sum_{i=1}^Ny_i(-2\Sigma^{-1}x_i+2\Sigma^{-1}\mu_1)=0 \\ &\iff \sum_{i=1}^Ny_i(\Sigma^{-1}\mu_1-\Sigma^{-1}x_i)=0 \\ &\iff \sum_{i=1}^Ny_i(\mu_1-x_i)=0 \\ &\iff \sum_{i=1}^Ny_i\mu_1=\sum_{i=1}^Ny_ix_i \\ &\iff \hat \mu_1=\frac{\sum\limits_{i=1}^Ny_ix_i}{\sum\limits_{i=1}^Ny_i}=\frac{\sum\limits_{i=1}^Ny_ix_i}{N_1} \\ \end{aligned}$
    同理可得
  $\hat \mu_2=\frac{\sum\limits_{i=1}^N(1-y_i)x_i}{\sum\limits_{i=1}^N(1-y_i)}=\frac{\sum\limits_{i=1}^N(1-y_i)x_i}{N_2}$
• 求 $\Sigma$:
    尝试对通项 $\log N(\mu,\Sigma)$进行化简，有
  $\begin{aligned} \sum_{i=1}^N\log N(\mu,\Sigma) &=\sum_{i=1}^N \log \frac{1}{(2\pi)^{\frac{p}{2}}|\Sigma|^{\frac{1}{2}}}\exp (-\frac{1}{2}(x_i-\mu)^T\Sigma^{-1}(x_i-\mu)) \\ &=\sum_{i=1}^N(\log \frac{1}{(2\pi)^{\frac{p}{2}}}+|\Sigma|^{-\frac{1}{2}}-\frac{1}{2}(x_i-\mu)^T\Sigma^{-1}(x_i-\mu)) \\ &=\sum_{i=1}^N(C-\frac{1}{2}\log|\Sigma|-\frac{1}{2}(x_i-\mu)^T\Sigma^{-1}(x_i-\mu))\\ &=C-\frac{1}{2}N\log |\Sigma|-\frac{1}{2}tr(\sum_{i=1}^N(x_i-\mu)^T\Sigma^{-1}(x_i-\mu))\\ &=C-\frac{1}{2}N\log |\Sigma|-\frac{1}{2}tr(\sum_{i=1}^N(x_i-\mu)(x_i-\mu)^T\Sigma^{-1})\\ &=-\frac{1}{2}N\log |\Sigma|-\frac{1}{2}tr(S\Sigma^{-1})+C\\ \end{aligned}$
    由于只需要对 $\Sigma$求解，所以对似然函数化简为
  $\begin{aligned} &\sum_{i=1}^N(y_i\log N(\mu_1,\Sigma) +(1-y_i)\log N(\mu_2,\Sigma) ) \\ &=\sum_{x_i \in c_1}\log N(\mu_1,\Sigma)+\sum_{x_i \in c_2}\log N(\mu_2,\Sigma) \\ &=-\frac{1}{2}N_1\log |\Sigma|-\frac{1}{2}tr(S_1\Sigma^{-1})-\frac{1}{2}N_2\log |\Sigma|-\frac{1}{2}N_2tr(S_2\Sigma^{-1})+C \\ &=-\frac{1}{2}(N_1\log |\Sigma|+N_1tr(S_1\Sigma^{-1})+N_2\log |\Sigma|+N_2tr(S_2\Sigma^{-1}))+C \\ \end{aligned}$
    根据tr的求导公式
  $\begin{aligned} &\frac{\partial tr(AB)}{\partial A}=B^{-1}\\ &\frac{\partial tr(|A|)}{\partial A}=|A|·A^{-1} \\ &tr(AB)=tr(BA) \end{aligned}$
    对上面化简后的式子进行求导并令导数为0，有
  $\begin{aligned} &-\frac{1}{2}(N\frac{1}{|\Sigma|}|\Sigma|\Sigma^{-1}+N_1\frac{\partial tr(\Sigma^{-1}S_1)}{\partial \Sigma^{-1}}\frac{\partial tr(\Sigma^{-1})}{\partial \Sigma}+N_2\frac{\partial tr(\Sigma^{-1}S_2)}{\partial \Sigma^{-1}}\frac{\partial tr(\Sigma^{-1})}{\partial \Sigma})=0 \\ &\iff N\frac{1}{|\Sigma|}|\Sigma|\Sigma^{-1}-N_1S_1^T\Sigma^{-2}-N_1S_2^T\Sigma^{-2}=0 \\ &\iff N\Sigma^{-1}-N_1S_1\Sigma^{-2}-N_1S_2\Sigma^{-2}=0\\ &\iff N\Sigma-N_1S_1-N_1S_2=0 \\ &\iff N\Sigma-N_1S_1-N_1S_2=0 \\ &\iff \hat \Sigma =\frac{1}{N}(N_1S_1+N_2S_2) \\ \end{aligned}$