\qquad
S
(
x
)
S(x)
S ( x )
S
(
x
)
=
1
1
+
e
−
x
\qquad\qquad S(x)=\dfrac{1}{1+e^{-x}}
S ( x ) = 1 + e − x 1
\qquad
[
S
(
x
)
]
′
=
S
(
x
)
[
1
−
S
(
x
)
]
[S(x) ]^{'}=S(x) [ 1-S(x) ]
[ S ( x ) ] ′ = S ( x ) [ 1 − S ( x ) ]
Sigmoid函数的曲线在中心
(
x
=
0
,
y
=
0.5
)
(x=0,y=0.5 )
( x = 0 , y = 0 . 5 )
\qquad
S
(
x
)
S(x)
S ( x )
f
(
x
)
=
w
T
x
+
b
f(\boldsymbol x)=\boldsymbol{w}^T \boldsymbol x + b
f ( x ) = w T x + b
y
(
x
)
=
S
[
f
(
x
)
]
=
1
1
+
e
−
(
w
T
x
+
b
)
\qquad\qquad y(\boldsymbol{x})=S [ f(\boldsymbol x) ]=\dfrac{1}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}
y ( x ) = S [ f ( x ) ] = 1 + e − ( w T x + b ) 1
\qquad
x
∗
\boldsymbol{x^{\ast}}
x ∗
y
=
y
(
x
∗
)
y=y(\boldsymbol x^{\ast})
y = y ( x ∗ )
ln
(
y
1
−
y
)
=
w
T
x
∗
+
b
\qquad\qquad \ln\left( \dfrac{y}{1-y}\right)=\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b
ln ( 1 − y y ) = w T x ∗ + b
\qquad
y
y
y
x
∗
\boldsymbol{x^{\ast}}
x ∗ 正例 的可能性（概率），将
1
−
y
1-y
1 − y
x
∗
\boldsymbol{x^{\ast}}
x ∗ 负例 的可能性（概率），两者的比率取对数
ln
(
y
1
−
y
)
\ln\left( \dfrac{y}{1-y}\right)
ln ( 1 − y y )
x
∗
\boldsymbol{x^{\ast}}
x ∗ 进行“线性分类”的情况 （如下图所示）：
1
)
\qquad1)
1 )
y
=
0.5
y=0.5
y = 0 . 5
1
−
y
=
0.5
1-y=0.5
1 − y = 0 . 5
ln
(
y
1
−
y
)
=
0
\ln\left( \dfrac{y}{1-y}\right)=0
ln ( 1 − y y ) = 0
w
T
x
∗
+
b
=
0
\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b=0
w T x ∗ + b = 0
\qquad
x
∗
\boldsymbol{x^{\ast}}
x ∗
2
)
\qquad2)
2 )
y
>
0.5
y>0.5
y > 0 . 5
1
−
y
<
0.5
1-y<0.5
1 − y < 0 . 5
ln
(
y
1
−
y
)
>
0
\ln\left( \dfrac{y}{1-y}\right)>0
ln ( 1 − y y ) > 0
w
T
x
∗
+
b
>
0
\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b>0
w T x ∗ + b > 0
\qquad
x
∗
\boldsymbol{x^{\ast}}
x ∗
3
)
\qquad3)
3 )
y
<
0.5
y<0.5
y < 0 . 5
1
−
y
>
0.5
1-y>0.5
1 − y > 0 . 5
ln
(
y
1
−
y
)
<
0
\ln\left( \dfrac{y}{1-y}\right)<0
ln ( 1 − y y ) < 0
w
T
x
∗
+
b
<
0
\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b<0
w T x ∗ + b < 0
\qquad
x
∗
\boldsymbol{x^{\ast}}
x ∗
通过Sigmoid函数可以将线性模型
y
=
w
T
x
∗
+
b
y=\boldsymbol{w}^{T}\boldsymbol{x^{\ast}}+b
y = w T x ∗ + b
y
y
y
[
0
,
1
]
[0,1]
[ 0 , 1 ]
p
p
p 几率 （odds）定义为
p
1
−
p
\dfrac{p}{1-p}
1 − p p 对数几率 （log odds）定义为
ln
(
p
1
−
p
)
\ln\left(\dfrac{p}{1-p}\right)
ln ( 1 − p p )
\qquad
c
=
1
c=1
c = 1
R
1
\mathcal R_1
R 1
c
=
0
c=0
c = 0
R
2
\mathcal R_2
R 2
y
(
x
)
y(\boldsymbol x)
y ( x ) 类后验概率 ，即：
p
(
c
=
1
∣
x
)
=
y
(
x
)
=
1
1
+
e
−
(
w
T
x
+
b
)
\qquad\qquad p(c=1|\boldsymbol{x})=y(\boldsymbol{x})=\dfrac{1}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}
p ( c = 1 ∣ x ) = y ( x ) = 1 + e − ( w T x + b ) 1
\qquad
p
(
c
=
1
∣
x
)
p(c=1|\boldsymbol{x})
p ( c = 1 ∣ x )
ln
p
(
c
=
1
∣
x
)
p
(
c
=
0
∣
x
)
=
w
T
x
+
b
\qquad\qquad\ln \dfrac{p(c=1|\boldsymbol{x})}{p(c=0|\boldsymbol{x})}=\boldsymbol{w}^T\boldsymbol{x}+b
ln p ( c = 0 ∣ x ) p ( c = 1 ∣ x ) = w T x + b
x
\boldsymbol{x}
x 正例
(
c
=
1
)
(c=1)
( c = 1 )
p
(
c
=
1
∣
x
)
=
1
1
+
e
−
(
w
T
x
+
b
)
\qquad\qquad p(c=1|\boldsymbol{x})=\dfrac{1}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}
p ( c = 1 ∣ x ) = 1 + e − ( w T x + b ) 1
\qquad
w
T
x
+
b
\boldsymbol{w}^{T}\boldsymbol{x}+b
w T x + b
+
∞
+\infty
+ ∞
1
1
1
\qquad
w
T
x
+
b
\boldsymbol{w}^{T}\boldsymbol{x}+b
w T x + b
−
∞
-\infty
− ∞
0
0
0
x
\boldsymbol{x}
x 负例
(
c
=
0
)
(c=0)
( c = 0 )
p
(
c
=
0
∣
x
)
=
1
−
p
(
y
=
1
∣
x
)
=
e
−
(
w
T
x
+
b
)
1
+
e
−
(
w
T
x
+
b
)
=
1
1
+
e
(
w
T
x
+
b
)
\qquad\qquad\begin{aligned} p(c=0|\boldsymbol{x})&=1-p(y=1|\boldsymbol{x})\\ &=\dfrac{e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}{1+e^{-(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}\\ &=\dfrac{1}{1+e^{(\boldsymbol{w}^{T}\boldsymbol{x}+b)}}\end{aligned}
p ( c = 0 ∣ x ) = 1 − p ( y = 1 ∣ x ) = 1 + e − ( w T x + b ) e − ( w T x + b ) = 1 + e ( w T x + b ) 1
\qquad
w
T
x
+
b
\boldsymbol{w}^{T}\boldsymbol{x}+b
w T x + b
+
∞
+\infty
+ ∞
0
0
0
\qquad
w
T
x
+
b
\boldsymbol{w}^{T}\boldsymbol{x}+b
w T x + b
−
∞
-\infty
− ∞
1
1
1
\qquad
x
∗
\boldsymbol{x^{\ast}}
x ∗
p
(
y
=
1
∣
x
∗
)
>
p
(
y
=
0
∣
x
∗
)
p(y=1|\boldsymbol{x^{\ast}})>p(y=0|\boldsymbol{x^{\ast}})
p ( y = 1 ∣ x ∗ ) > p ( y = 0 ∣ x ∗ )
x
∗
\boldsymbol{x^{\ast}}
x ∗
R
1
R_{1}
R 1
p
(
y
=
1
∣
x
∗
)
<
p
(
y
=
0
∣
x
∗
)
p(y=1|\boldsymbol{x^{\ast}})<p(y=0|\boldsymbol{x^{\ast}})
p ( y = 1 ∣ x ∗ ) < p ( y = 0 ∣ x ∗ )
x
∗
\boldsymbol{x^{\ast}}
x ∗
R
2
R_{2}
R 2
\qquad
\qquad
{
(
x
i
,
c
i
)
}
i
=
1
N
\{ ( \boldsymbol{x}_{i},c_{i}) \} _{i=1}^N
{ ( x i , c i ) } i = 1 N
x
i
∈
R
n
,
c
i
∈
{
0
,
1
}
\boldsymbol{x}_{i}\in R^{n},c_{i}\in \{0,1\}
x i ∈ R n , c i ∈ { 0 , 1 }
(
w
,
b
)
(\boldsymbol{w},b)
( w , b )
1
)
\qquad1)
1 )
y
(
x
)
=
p
(
c
=
1
∣
x
)
y(\boldsymbol{x})=p(c=1|\boldsymbol{x})
y ( x ) = p ( c = 1 ∣ x )
L
(
w
,
b
)
=
∏
i
=
1
N
y
(
x
i
)
c
i
[
1
−
y
(
x
i
)
]
1
−
c
i
\qquad\qquad L(\boldsymbol{w},b)=\displaystyle\prod_{i=1}^N y(\boldsymbol{x}_{i})^{c_{i}}\left[ 1-y(\boldsymbol{x}_{i})\right] ^{1-c_{i}}
L ( w , b ) = i = 1 ∏ N y ( x i ) c i [ 1 − y ( x i ) ] 1 − c i
2
)
\qquad2)
2 )
ln
L
(
w
,
b
)
=
∑
i
=
1
N
{
c
i
ln
[
y
(
x
i
)
]
+
(
1
−
c
i
)
ln
[
1
−
y
(
x
i
)
]
}
=
∑
i
=
1
N
{
c
i
ln
y
(
x
i
)
1
−
y
(
x
i
)
+
ln
[
1
−
y
(
x
i
)
]
}
=
∑
i
=
1
N
{
c
i
(
w
T
x
i
+
b
)
−
ln
[
1
+
e
(
w
T
x
i
+
b
)
]
}
\qquad\qquad\begin{aligned} \ln L(\boldsymbol{w},b)&=\displaystyle\sum_{i=1}^N \{ c_{i}\ln\left[ y\left( \boldsymbol{x}_{i}\right) \right] +(1-c_{i})\ln\left[ 1-y\left( \boldsymbol{x}_{i}\right) \right] \}\\ &=\displaystyle\sum_{i=1}^N\left\{ c_{i}\ln\dfrac{ y\left( \boldsymbol{x}_{i}\right)}{1-y\left( \boldsymbol{x}_{i}\right)} +\ln\left[ 1-y\left( \boldsymbol{x}_{i}\right) \right] \right\} \\ &=\displaystyle\sum_{i=1}^N\left\{ c_{i}\left(\boldsymbol{w}^{T}\boldsymbol{x}_{i}+b\right)-\ln\left[ 1+e^{\left(\boldsymbol{w}^{T}\boldsymbol{x}_{i}+b\right)} \right] \right\} \\ \end{aligned}
ln L ( w , b ) = i = 1 ∑ N { c i ln [ y ( x i ) ] + ( 1 − c i ) ln [ 1 − y ( x i ) ] } = i = 1 ∑ N { c i ln 1 − y ( x i ) y ( x i ) + ln [ 1 − y ( x i ) ] } = i = 1 ∑ N { c i ( w T x i + b ) − ln [ 1 + e ( w T x i + b ) ] }
3
)
\qquad3)
3 )
β
=
[
w
T
,
b
]
T
,
x
^
i
=
[
x
i
T
,
1
]
T
\boldsymbol{\beta}=[\boldsymbol{w}^T,b]^{T},\ \hat{\boldsymbol{x}}_{i}=[\boldsymbol{x}_{i}^T,1]^{T}
β = [ w T , b ] T , x ^ i = [ x i T , 1 ] T
w
T
x
i
+
b
=
β
T
x
^
i
\boldsymbol{w}^{T}\boldsymbol{x}_{i}+b=\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}
w T x i + b = β T x ^ i
ln
L
(
β
)
=
∑
i
=
1
N
[
c
i
β
T
x
^
i
−
ln
(
1
+
e
β
T
x
^
i
)
]
\qquad\qquad \ln L(\boldsymbol{\beta})=\displaystyle\sum_{i=1}^N \left[ c_{i}\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}-\ln\left( 1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}} \right) \right]
ln L ( β ) = i = 1 ∑ N [ c i β T x ^ i − ln ( 1 + e β T x ^ i ) ]
\qquad
β
=
[
w
T
,
b
]
T
\boldsymbol{\beta}=[\boldsymbol{w}^{T},b]^{T}
β = [ w T , b ] T
\qquad
l
(
β
)
=
−
ln
L
(
w
,
b
)
=
−
ln
L
(
β
)
l(\boldsymbol{\beta})=-\ln L(\boldsymbol{w},b)=-\ln L(\boldsymbol{\beta})
l ( β ) = − ln L ( w , b ) = − ln L ( β )
l
(
β
)
l(\boldsymbol{\beta})
l ( β )
l
(
β
)
=
−
ln
L
(
β
)
=
−
∑
i
=
1
N
[
c
i
β
T
x
^
i
−
ln
(
1
+
e
β
T
x
^
i
)
]
\qquad\qquad l(\boldsymbol{\beta})=-\ln L(\boldsymbol{\beta})=-\displaystyle\sum_{i=1}^N \left[ c_{i}\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}-\ln\left( 1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}} \right) \right]
l ( β ) = − ln L ( β ) = − i = 1 ∑ N [ c i β T x ^ i − ln ( 1 + e β T x ^ i ) ]
\qquad
l
(
β
)
l(\boldsymbol{\beta})
l ( β )
β
\boldsymbol{\beta}
β
β
\boldsymbol{\beta}
β
\qquad
\qquad
∂
l
(
β
)
∂
β
=
−
∑
i
=
1
N
(
c
i
x
^
i
−
1
1
+
e
β
T
x
^
i
e
β
T
x
^
i
x
^
i
)
=
−
∑
i
=
1
N
(
c
i
−
e
β
T
x
^
i
1
+
e
β
T
x
^
i
)
x
^
i
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
x
^
i
(
1
)
\qquad\qquad \begin{aligned} \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}&=-\displaystyle\sum_{i=1}^N \left( c_{i}\hat{\boldsymbol{x}}_{i}-\dfrac{1}{1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}}e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}\hat{\boldsymbol{x}}_{i} \right)\\ &=-\displaystyle\sum_{i=1}^N \left( c_{i}-\dfrac{e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}}{1+e^{\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}}} \right)\hat{\boldsymbol{x}}_{i} \\ &=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i} \qquad\qquad\qquad\ (1)\\ \end{aligned}
∂ β ∂ l ( β ) = − i = 1 ∑ N ( c i x ^ i − 1 + e β T x ^ i 1 e β T x ^ i x ^ i ) = − i = 1 ∑ N ( c i − 1 + e β T x ^ i e β T x ^ i ) x ^ i = − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i ( 1 )
\qquad
β
=
[
w
T
,
b
]
T
=
[
w
1
,
⋯
,
w
n
,
b
]
T
\boldsymbol{\beta}=[\boldsymbol{w}^T,b]^{T}=[w_1,\cdots,w_n,b]^T
β = [ w T , b ] T = [ w 1 , ⋯ , w n , b ] T
x
^
i
=
[
x
i
T
,
1
]
T
=
[
x
i
(
1
)
,
⋯
,
x
i
(
n
)
,
1
]
T
\hat\boldsymbol{x}_{i}=[\boldsymbol{x}_{i}^T,1]^T=[x_i^{(1)},\cdots,x_i^{(n)},1]^T
x ^ i = [ x i T , 1 ] T = [ x i ( 1 ) , ⋯ , x i ( n ) , 1 ] T
{
∂
l
(
β
)
∂
w
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
x
i
(
2
)
∂
l
(
β
)
∂
b
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
(
3
)
\qquad\qquad\begin{cases}\ \ \ \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol w}=-\displaystyle\sum_{i=1}^N [ c_{i}-y\left( \boldsymbol{x}_{i}\right) ]\boldsymbol{x}_{i}\qquad\qquad\ \ (2) \\ \\ \ \ \ \dfrac{\partial l(\boldsymbol{\beta})}{\partial b}=-\displaystyle\sum_{i=1}^N\left[ c_{i}-y\left( \boldsymbol{x}_{i}\right) \right] \qquad\qquad(3) \end{cases}
⎩ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎧ ∂ w ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x i ( 2 ) ∂ b ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] ( 3 )
\qquad
x
i
=
[
x
i
(
1
)
,
⋯
,
x
i
(
n
)
]
T
\boldsymbol{x}_{i}=[x_i^{(1)},\cdots,x_i^{(n)}]^T
x i = [ x i ( 1 ) , ⋯ , x i ( n ) ] T
x
i
(
j
)
x_{i}^{(j)}
x i ( j )
∂
l
(
β
)
∂
w
j
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
x
i
(
j
)
\qquad\qquad \dfrac{\partial l(\boldsymbol{\beta})}{\partial w_{j}}=-\displaystyle\sum_{i=1}^N [ c_{i}-y\left( \boldsymbol{x}_{i}\right) ]x_{i}^{(j)}
∂ w j ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x i ( j )
\qquad
∂
l
(
β
)
∂
β
=
[
∂
l
(
β
)
∂
w
1
,
⋯
,
∂
l
(
β
)
∂
w
n
,
∂
l
(
β
)
∂
b
]
T
\qquad\qquad \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=\left[\dfrac{\partial l(\boldsymbol{\beta})}{\partial w_1},\cdots,\dfrac{\partial l(\boldsymbol{\beta})}{\partial w_n},\dfrac{\partial l(\boldsymbol{\beta})}{\partial b} \right]^{T}
∂ β ∂ l ( β ) = [ ∂ w 1 ∂ l ( β ) , ⋯ , ∂ w n ∂ l ( β ) , ∂ b ∂ l ( β ) ] T
\qquad
α
\alpha
α
β
t
+
1
=
β
t
−
α
∂
l
(
β
)
∂
β
\qquad\qquad \boldsymbol{\beta}^{t+1}=\boldsymbol{\beta}^{t}-\alpha\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}
β t + 1 = β t − α ∂ β ∂ l ( β )
\qquad
\qquad
0
0
0
h
e
s
s
i
a
n
hessian
h e s s i a n
\qquad
∂
l
(
β
)
∂
β
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
x
^
i
\ \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i}
∂ β ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i
\qquad
h
e
s
s
i
a
n
hessian
h e s s i a n
∂
∂
β
T
(
∂
l
(
β
)
∂
β
)
=
∂
∂
β
T
(
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
x
^
i
)
=
∂
∂
β
T
(
∑
i
=
1
N
y
(
x
i
)
x
^
i
)
=
∑
i
=
1
N
y
(
x
i
)
[
1
−
y
(
x
i
)
]
x
^
i
∂
∂
β
T
(
β
T
x
^
i
)
=
∑
i
=
1
N
y
(
x
i
)
[
1
−
y
(
x
i
)
]
x
^
i
x
^
i
T
\qquad\qquad \begin{aligned} \dfrac{\partial}{\partial \boldsymbol\beta^T}\left(\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol\beta}\right) &=\dfrac{\partial}{\partial \boldsymbol\beta^T}\left(-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i}\right)\\ &=\dfrac{\partial}{\partial \boldsymbol\beta^T}\left(\displaystyle\sum_{i=1}^Ny(\boldsymbol{x}_{i})\hat\boldsymbol{x}_i\right)\\ &=\displaystyle\sum_{i=1}^Ny(\boldsymbol{x}_{i})[1-y(\boldsymbol{x}_{i})]\hat\boldsymbol{x}_i\dfrac{\partial}{\partial \boldsymbol\beta^T}\left(\boldsymbol{\beta}^{T}\hat{\boldsymbol{x}}_{i}\right)\\ &=\displaystyle\sum_{i=1}^Ny(\boldsymbol{x}_{i})[1-y(\boldsymbol{x}_{i})]\hat\boldsymbol{x}_i\hat{\boldsymbol{x}}_{i}^T\\ \end{aligned}
∂ β T ∂ ( ∂ β ∂ l ( β ) ) = ∂ β T ∂ ( − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i ) = ∂ β T ∂ ( i = 1 ∑ N y ( x i ) x ^ i ) = i = 1 ∑ N y ( x i ) [ 1 − y ( x i ) ] x ^ i ∂ β T ∂ ( β T x ^ i ) = i = 1 ∑ N y ( x i ) [ 1 − y ( x i ) ] x ^ i x ^ i T
\qquad
β
t
+
1
=
β
t
−
(
∂
2
l
(
β
)
∂
β
∂
β
T
)
−
1
∂
l
(
β
)
∂
β
\qquad\qquad \boldsymbol{\beta}^{t+1}=\boldsymbol{\beta}^{t}-\left(\dfrac{\partial^2 l(\boldsymbol\beta)}{\partial \boldsymbol\beta\partial \beta^T}\right)^{-1}\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}
β t + 1 = β t − ( ∂ β ∂ β T ∂ 2 l ( β ) ) − 1 ∂ β ∂ l ( β )
\qquad
\qquad
1
)
\qquad1)
1 )
β
=
[
w
T
,
b
]
T
\boldsymbol{\beta}=[\boldsymbol{w}^T,b]^{T}
β = [ w T , b ] T
β
0
\boldsymbol{\beta}^{0}
β 0
2
)
\qquad2)
2 )
α
\alpha
α
∂
l
(
β
)
∂
β
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
x
^
i
=
−
∑
i
=
1
N
[
c
i
−
y
(
x
i
)
]
[
x
i
1
]
\qquad\qquad \begin{aligned} \dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}} &=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\hat{\boldsymbol{x}}_{i}\\ &=-\displaystyle\sum_{i=1}^N [ c_{i}-y(\boldsymbol{x}_{i}) ]\left[\begin{matrix}\boldsymbol{x}_{i}\\ 1\end{matrix}\right]\\ \end{aligned}
∂ β ∂ l ( β ) = − i = 1 ∑ N [ c i − y ( x i ) ] x ^ i = − i = 1 ∑ N [ c i − y ( x i ) ] [ x i 1 ]
β
t
+
1
=
β
t
−
α
∂
l
(
β
)
∂
β
\qquad\qquad \boldsymbol{\beta}^{t+1}=\boldsymbol{\beta}^{t}-\alpha\dfrac{\partial l(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}
β t + 1 = β t − α ∂ β ∂ l ( β )
\qquad
1) 定义Sigmoid函数
def sigmoid ( x) :
'''Sigmoid function
'''
return 1.0 / ( 1 + np. exp( - x) )
2) 读取训练/测试集数据函数
假设在二维平面
R
2
R^{2}
R 2
(
x
i
,
y
i
)
=
(
x
i
(
1
)
,
x
i
(
2
)
,
y
i
)
,
y
i
∈
{
0
,
1
}
(\boldsymbol{x}_{i},y_{i})=(x_{i}^{(1)},x_{i}^{(2)},y_{i}), \ y_{i}\in\{0,1\}
( x i , y i ) = ( x i ( 1 ) , x i ( 2 ) , y i ) , y i ∈ { 0 , 1 }
3.562302
,
25.329208
,
1.000000
−
24.268267
,
1.272092
,
1.000000
25.405790
,
8.463017
,
1.000000
−
6.908775
,
23.298889
,
1.000000
40.621010
,
−
25.134052
,
0.000000
−
9.305521
,
14.983097
,
1.000000
20.041330
,
−
25.381725
,
0.000000
37.298540
,
−
26.767307
,
0.000000
35.856177
,
−
31.080316
,
0.000000
−
17.976889
,
4.244106
,
1.000000
⋅
⋅
⋅
⋅
⋅
⋅
3.562302,25.329208,1.000000\newline -24.268267,1.272092,1.000000\newline 25.405790,8.463017,1.000000\newline -6.908775,23.298889,1.000000\newline 40.621010,-25.134052,0.000000\newline -9.305521,14.983097,1.000000\newline 20.041330,-25.381725,0.000000\newline 37.298540,-26.767307,0.000000\newline 35.856177,-31.080316,0.000000\newline -17.976889,4.244106,1.000000\newline \cdot\cdot\cdot\cdot\cdot\cdot
3 . 5 6 2 3 0 2 , 2 5 . 3 2 9 2 0 8 , 1 . 0 0 0 0 0 0 − 2 4 . 2 6 8 2 6 7 , 1 . 2 7 2 0 9 2 , 1 . 0 0 0 0 0 0 2 5 . 4 0 5 7 9 0 , 8 . 4 6 3 0 1 7 , 1 . 0 0 0 0 0 0 − 6 . 9 0 8 7 7 5 , 2 3 . 2 9 8 8 8 9 , 1 . 0 0 0 0 0 0 4 0 . 6 2 1 0 1 0 , − 2 5 . 1 3 4 0 5 2 , 0 . 0 0 0 0 0 0 − 9 . 3 0 5 5 2 1 , 1 4 . 9 8 3 0 9 7 , 1 . 0 0 0 0 0 0 2 0 . 0 4 1 3 3 0 , − 2 5 . 3 8 1 7 2 5 , 0 . 0 0 0 0 0 0 3 7 . 2 9 8 5 4 0 , − 2 6 . 7 6 7 3 0 7 , 0 . 0 0 0 0 0 0 3 5 . 8 5 6 1 7 7 , − 3 1 . 0 8 0 3 1 6 , 0 . 0 0 0 0 0 0 − 1 7 . 9 7 6 8 8 9 , 4 . 2 4 4 1 0 6 , 1 . 0 0 0 0 0 0 ⋅ ⋅ ⋅ ⋅ ⋅ ⋅
生成具有两个中心的二维高斯分布的数据集：
def gen_gausssian ( mean1, mean2, cov1, cov2, num) :
'''generate 2-d gaussian dataset with 2 clusters
'''
data1 = np. random. multivariate_normal( mean1, cov1, num)
label1 = np. ones( ( 1 , num) ) . T
data_pos = np. append( data1, label1, axis= 1 )
data2 = np. random. multivariate_normal( mean2, cov2, num)
label2 = np. zeros( ( 1 , num) ) . T
data_neg = np. append( data2, label2, axis= 1 )
data = np. append( data_pos, data_neg, axis= 0 )
shuffle_data = np. random. permutation( data)
x1, y1 = data1. T
x2, y2 = data2. T
plt. scatter( x1, y1, c= 'r' , s= 3 )
plt. plot( mean1[ 0 ] , mean1[ 1 ] , 'ko' )
plt. scatter( x2, y2, c= 'b' , s= 3 )
plt. plot( mean2[ 0 ] , mean2[ 1 ] , 'ko' )
plt. axis( )
plt. title( "2-d gaussian dataset with 2 clusters" )
plt. xlabel( "x" )
plt. ylabel( "y" )
plt. show( )
np. savetxt( 'gaussdata.txt' , shuffle_data, fmt= '%f' , delimiter= ',' )
return shuffle_data, data_pos, data_neg
用散点图表示：
(
x
i
,
y
i
)
=
(
x
i
(
1
)
,
x
i
(
2
)
,
y
i
)
,
y
i
∈
{
0
,
1
}
(\boldsymbol{x}_{i},y_{i})=(x_{i}^{(1)},x_{i}^{(2)},y_{i}), \ y_{i}\in\{0,1\}
( x i , y i ) = ( x i ( 1 ) , x i ( 2 ) , y i ) , y i ∈ { 0 , 1 }
def load_data ( filename) :
'''Load data of training or testing set
'''
tdata = [ ]
with open ( filename) as f:
while True :
line = f. readline( )
if not line:
break
line = line. split( ',' )
tdata. append( [ float ( item) for item in line] )
f. close( )
return np. array( tdata)
3) 迭代计算公式
(
2
)
(2)
( 2 )
def lr_train ( xhat, c, alpha, num) :
beta = np. random. rand( 3 , 1 )
for i in range ( num) :
yx = sigmoid( np. dot( xhat, beta) )
beta = beta + alpha * np. dot( xhat. T, ( c - yx) )
print ( '#' + str ( i) + ',training loss:' + str ( train_loss( c, yx) ) )
return beta
由公式
−
ln
L
(
w
,
b
)
=
−
∑
i
=
1
N
{
c
i
ln
[
y
(
x
i
)
]
+
(
1
−
c
i
)
ln
[
1
−
y
(
x
i
)
]
}
\ -\ln L(\boldsymbol{w},b)=-\displaystyle\sum_{i=1}^N \{ c_{i}\ln\left[ y\left( \boldsymbol{x}_{i}\right) \right] +(1-c_{i})\ln\left[ 1-y\left( \boldsymbol{x}_{i}\right) \right] \}
− ln L ( w , b ) = − i = 1 ∑ N { c i ln [ y ( x i ) ] + ( 1 − c i ) ln [ 1 − y ( x i ) ] }
计算损失函数（误差值） ：
def train_loss ( c, yx) :
err = 0.0
for i in range ( len ( yx) ) :
if yx[ i, 0 ] > 0 and ( 1 - yx[ i, 0 ] ) > 0 :
err -= c[ i, 0 ] * np. log( yx[ i, 0 ] ) + ( 1 - c[ i, 0 ] ) * np. log( 1 - yx[ i, 0 ] )
return err
主程序：
mean1 = [ 3 , - 1 ]
cov1 = [ [ 5 , 0 ] , [ 0 , 10 ] ]
mean2 = [ - 5 , 7 ]
cov2 = [ [ 10 , 0 ] , [ 0 , 5 ] ]
data, data_pos, data_neg = gen_gausssian( mean1, mean2, cov1, cov2, 1100 )
training_data = data
tmp1 = training_data[ 0 : 2000 , 0 : 2 ]
tmp2 = np. ones( ( 2000 , 1 ) )
xhat = np. concatenate( ( tmp1, tmp2) , axis= 1 )
target = training_data[ 0 : 2000 , 2 : ]
beta = lr_train( xhat, target, 0.01 , 100 )
print ( 'beta:\n' , beta)
tmp1 = training_data[ 2000 : 2200 , 0 : 2 ]
tmp2 = np. ones( ( 200 , 1 ) )
testing_data = np. concatenate( ( tmp1, tmp2) , axis= 1 )
target = training_data[ 2000 : 2200 , 2 : ]
y1 = classification( testing_data, beta)
print ( np. abs ( y1- target) . T)
对一个大小为2000的数据进行训练，可得到输出结果为：
beta:
这里的beta值就是用梯度下降法所求得
β
=
[
w
T
,
b
]
T
\boldsymbol{\beta}=[\boldsymbol{w}^{T},b]^{T}
β = [ w T , b ] T
def classification ( testing_data, beta) :
y = sigmoid( np. dot( testing_data, beta) )
for i in range ( len ( y) ) :
if y[ i, 0 ] < 0.5 :
y[ i, 0 ] = 0.0
else :
y[ i, 0 ] = 1.0
return y
判别结果：200个测试数据，有2个数据被错误分类
