3.2 试证明，对于参数$\boldsymbol \omega$，，对率回归的目标函数(3.18)是非凸的，但其对数似然函数(3.27)是凸的。
解答：

定理：设$f(\boldsymbol x)$是定义在非空开集$D\subset \mathbb R^n$上的二次可微函数，则$f(\boldsymbol x)$是凸函数的充要条件是在任意点$\boldsymbol x\in D$处，$f(\boldsymbol x)$的Hessian矩阵半正定。

无论是目标函数，还是对数似然函数，二次可微的条件是成立的。所以，只需要判断函数是否满足在任意点处的Hessian矩阵半正定。

$$y = \frac{1}{1+e^{-(\boldsymbol \omega^T\boldsymbol x+b)}}\tag 1$$
$$\frac{\partial y}{\partial \boldsymbol \omega}=\frac{e^{-(\boldsymbol \omega^T\boldsymbol x+b)}}{(1+e^{-(\boldsymbol \omega^T\boldsymbol x+b)} )^2}\boldsymbol x \tag 2$$
$$\begin{align} \frac{\partial^2 y}{\partial \boldsymbol \omega \partial \boldsymbol \omega^T }&=\frac{\partial}{\partial \boldsymbol \omega^T}\frac{\partial y}{\partial \boldsymbol \omega} \\ &=\frac{\partial}{\partial \boldsymbol \omega^T}\frac{e^{-(\boldsymbol \omega^T\boldsymbol x+b)}}{(1+e^{-(\boldsymbol \omega^T\boldsymbol x+b)} )^2}\boldsymbol x \\ & = \frac{e^{-(\boldsymbol \omega^T\boldsymbol x+b)}(1-e^{-(\boldsymbol \omega^T\boldsymbol x+b)})}{(1+e^{-(\boldsymbol \omega^T\boldsymbol x+b)})^3}\boldsymbol x \boldsymbol x^T \\ &=y(1-y)(1-2y)\boldsymbol x \boldsymbol x^T \tag 3 \end{align}$$
矩阵 $\boldsymbol x \boldsymbol x^T$是半正定矩阵。而 $y(1-y)(1-2y)$在 $y\in(\frac{1}{2},1)$上是小于0的。所以Hessian矩阵并不能保证总是非负的，即函数(1)是非凸的。

$$l=\sum_{i=1}^m\left( -y_i\boldsymbol \beta^T \boldsymbol{\hat x}_i+ \ln\left( 1+e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}\right) \right) \tag 4$$
$$\frac{\partial l}{\partial \boldsymbol\beta}=\sum_{i=1}^{m}\left( -y_i\boldsymbol{\hat x}_i +\frac{e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}}{1+e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}} \boldsymbol{\hat x}_i \right) \tag 5$$
$$\begin{align} \frac{\partial^2 l}{\partial\boldsymbol\beta \partial \boldsymbol \beta ^T }&=\frac{\partial }{\partial \boldsymbol \beta^T}\frac{\partial l}{\partial \boldsymbol \beta} \\ &=\frac{\partial }{\partial \boldsymbol \beta^T}\sum_{i=1}^{m}\left( -y_i\boldsymbol{\hat x}_i +\frac{e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}}{1+e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}} \boldsymbol{\hat x}_i \right) \\ &=\sum_{i=1}^{m}\frac{e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}}{(1+e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i})^2} \boldsymbol{\hat x}_i\boldsymbol{\hat x}_i^T \end{align} \tag 6$$
因为 $\frac{e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i}}{(1+e^{\boldsymbol\beta^T\boldsymbol{\hat x}_i})^2}>0$，矩阵 $\boldsymbol{\hat x}_i\boldsymbol{\hat x}_i^T$半正定，所以 $\frac{\partial^2 l}{\partial\boldsymbol\beta \partial \boldsymbol \beta ^T }$也是半正定的，所以函数(4)是凸函数。

3.3 西瓜书《机器学习》课后答案——Chapter3_3.3

3.4 西瓜书《机器学习》课后答案——Chapter3_3.4

3.5 西瓜书《机器学习》课后答案——Chapter3_3.5