BatchNormal推导和yolo源码解析

最近一直在看yolo算法,感觉有时候只有看懂各种算法底层的实现,才能真正了解这个算法,这不,结合yolo源码和网上各种算法讲解,各种深度算法不再是模糊的印象了,而这里总结了最近看的BatchNormal

1.batchnormal解决了什么问题

1)梯度消失问题
对于网络任一层l,输入 $a^{l-1}$ ,一般都会经历一下过程:
$z=wa^{l-1}+b$
输出: $a^l=\sigma(z)$ ,
那么问题来了,假设激活函数 $\sigma$ 为逻辑函数:

如果某层的z分布在逻辑函数两边的接近水平的地方,也就是梯度接近0的位置,那么网络反向传播这层后,梯度降为接近0,梯度消失,那么参数,也就基本不更新了.
这里写图片描述

2)参数0均值初始化导致的问题
由于初始化的时候，参数一般都是0均值的，因此开始的拟合y=Wx+b，基本过原点附近，如图b红色虚线。因此，网络需要经过多次学习才能逐步达到如紫色实线的拟合，即收敛的比较慢。如果我们对输入数据先作减均值操作，如图c，显然可以加快学习。更进一步的，我们对数据再进行去相关操作，使得数据更加容易区分，这样又会加快训练，如图d。而batchnormal也有这种效果

3)内部迁移(Internal Covariate Shift)
数据经过多层神经网络后,数据分布发生变化,导致各层参数需要不到调整适应分布变化,这会让收敛速度变慢,超参数设定也变得比较复杂,这在论文里作者称作Internal Covariate Shift

那么batchnormal具体如何解决上述的呢,首先了解前行传播

`注意以下字母皆为矩阵形式,i为batch里面的索引`

2.BN前向传播

这里写图片描述
设有 $mini$ _ $batch$ 的batch_size为m,注意这里的i为batch里面的索引,x和a为特征矩阵
由于当前层的输入等于上一层的输出,那么设第l层输出为 $a^l$ ,则第l层的输入 $x^l_i=a^{l-1}_i,$ i =0,1,2…m
设

前向传播过程:
1.全连接则乘权重,卷积层则对x卷积

\begin{matrix} (1.1) & z_{i}^{l} = (w_{i}^{l})^{T} a_{i}^{l - 1} / / 偏 重 放 后 面 加 \end{matrix}

$z^l_i=(w^l_i)^Ta^{l-1}_i\tag{1.1}\quad//偏重放后面加$
2.计算batch_size个z的均值

\begin{matrix} (1.2) & μ_{B}^{l} = \frac{1}{m} \sum_{i = 1}^{m} z_{i}^{l} \end{matrix}

$\mu^l_B=\frac 1 m \sum_{i=1}^{m}{z_i^l} \tag{1.2}$
3)计算batch_size个z的方差:

\begin{matrix} (1.3) & (D_{B}^{2})^{(l)} = \frac{1}{m} \sum_{i = 1}^{m} (z_{i}^{l} - μ_{B}^{l})^{2} \end{matrix}

$(D_B^2)^{(l)}=\frac1 m\sum_{i=1}^m{(z^l_i-\mu^l_B)^2}\tag{1.3}$
3)将batch_size个z,归一化成均值为0,方差为1的分布:

\begin{matrix} (1.4) & B N = {\hat{z}}_{i}^{l} = \frac{z_{i}^{l} - μ_{B}^{l}}{\sqrt{(D_{B}^{2})^{(l)} + ε}} \end{matrix}

$BN=\hat{z}^l_i=\frac {z^l_i-\mu^l_B}{\sqrt{(D_B^2)^{(l)}+\varepsilon}}\tag{1.4}$

经过这个操作后数据就被分布在0为圆心,1为半径的范围内了,这样以上问题就被成功解决了问题
这里写图片描述

4)放缩和迁移:

\begin{matrix} (1.5) & y_{i}^{l} = γ {\hat{z}}_{i}^{l} + β \end{matrix}

$y^l_i=\gamma \hat{z}^l_i+\beta \tag{1.5}$
这步的作用在于,以逻辑函数为例,经过1.3后,数据主要分布在线性区域,非线性表达能力会受到影响,所以通过对数据放大或缩小和迁移来进入非线性区域范围
5)激活,输出

\begin{matrix} (1.6) & a_{i}^{l} = σ (y_{i}^{l}) \end{matrix}

$a^l_i=\sigma(y^l_i)\tag{1.6}$

yolo的前行传播源码：

void forward_batchnorm_layer(layer l, network net)
{
    //如果是batchnormal层,则直接输出等于输入
    if(l.type == BATCHNORM) copy_cpu(l.outputs*l.batch, net.input, 1, l.output, 1);
    //全链接层，看成通道数为l.outputs,特征图长宽为1
    if(l.type == CONNECTED){
        l.out_c = l.outputs;
        l.out_h = l.out_w = 1;
    }

    //l.x=l.output,如果按方差求导最终的化简等于0,来计算,l.x后面就用不到了
    copy_cpu(l.outputs*l.batch, l.output, 1, l.x, 1);

    //训练状态
    if(net.train){
        //求当前batch的均值，对应公式1.2
        mean_cpu(l.output, l.batch, l.out_c, l.out_h*l.out_w, l.mean);
        //求当前batch的方差，对应公式1.3
        variance_cpu(l.output, l.mean, l.batch, l.out_c, l.out_h*l.out_w, l.variance);
        //求均值的滚动平均，预测时,均值的就是这个值,什么是滚动平均，见下面注
        scal_cpu(l.out_c, .99, l.rolling_mean, 1);
        axpy_cpu(l.out_c, .01, l.mean, 1, l.rolling_mean, 1);

        //求方差的滚动平均,预测时,方差用的就是这个值,可以看非训练状态时normalize_cpu()函数的实现和参数
        scal_cpu(l.out_c, .99, l.rolling_variance, 1);
        axpy_cpu(l.out_c, .01, l.variance, 1, l.rolling_variance, 1);

        //对应公式1.4
        normalize_cpu(l.output, l.mean, l.variance, l.batch, l.out_c, l.out_h*l.out_w);   

        //将1.4式子的结果保存到l.x_norm,用于反向传播时相关参数梯度的计算
        copy_cpu(l.outputs*l.batch, l.output, 1, l.x_norm, 1);
    } 
    //非训练状态，如预测时
    else {
        //对应公式1.4
        normalize_cpu(l.output, l.rolling_mean, l.rolling_variance, l.batch, l.out_c, l.out_h*l.out_w);
    }
    //这两步,对应公式1.5,这里l.scale对应gamma,l.biases对应beta
    scale_bias(l.output, l.scales, l.batch, l.out_c, l.out_h*l.out_w);
    add_bias(l.output, l.biases, l.batch, l.out_c, l.out_h*l.out_w);
}

注：
滚动平均：

\begin{aligned} {\bar{x}}_{n} & = \frac{x_{1} + x_{2} + \cdot \cdot \cdot + x_{m}}{m} \\ = \frac{x_{1} + x_{2} + \cdot \cdot \cdot + x_{m - 1} + x_{m}}{m} \\ = \frac{x_{1} + x_{2} + \cdot \cdot \cdot + x_{m - 1}}{m} + \frac{x_{m}}{m} \\ = \frac{(x_{1} + x_{2} + \cdot \cdot \cdot + x_{m - 1}) (m - 1)}{m (m - 1)} + \frac{x_{m}}{m} \\ = \frac{m - 1}{m} {\bar{x}}_{m - 1} + \frac{x_{m}}{m} \\ = (1 - \frac{1}{m}) {\bar{x}}_{m - 1} + \frac{1}{m} x_{m} \end{aligned}

$\begin{align} \bar x_n&=\frac {x_1+x_2+\cdot\cdot\cdot+x_m}{m}\nonumber \\ &=\frac{x_1+x_2+\cdot\cdot\cdot+x_{m-1}+x_m}{m}\nonumber \\ &=\frac{x_1+x_2+\cdot\cdot\cdot+x_{m-1}}{m}+\frac{x_m}{m}\nonumber \\ &=\frac{(x_1+x_2+\cdot\cdot\cdot+x_{m-1})(m-1)}{m(m-1)}+\frac{x_m}{m}\nonumber \\ &=\frac{m-1}{m}\bar x_{m-1}+\frac {x_m}{m}\nonumber \\ &=(1-\frac 1 m)\bar x_{m-1}+\frac 1 mx_m \nonumber \\ \end{align}$

这里 $\bar x_n$ 表示前n个数据的平均值
yolo在这里直接将 $\frac 1 n=0.1$ ,简化计算,
至于为什么可以用这个均值来近似代替整个数据集的分分布,见后面的预测解释,推导

3.BN反向传播过程

首先推导几个值,为后面链式求导用:
设最终的损失函数为 $C$
对方差求导:

\begin{aligned} \frac{\partial C}{\partial (D_{B}^{2})^{l}} & = \sum_{i = 1}^{m} (\frac{\partial C}{\partial {\hat{z}}_{i}^{l}} \frac{\partial {\hat{z}}_{i}^{l}}{\partial (D_{B}^{2})^{l}}) \\ = \sum_{i = 1}^{m} {\frac{- 1}{2} \frac{\partial C}{\partial z_{i}^{l + 1}} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} \frac{\partial a_{i}^{l}}{\partial y_{i}^{l}} \frac{\partial y_{i}^{l}}{\partial {\hat{z}}_{i}^{l}} (z_{i}^{l} - μ_{B}^{l}) [(D_{B}^{2})^{(l)} + ε]^{- \frac{3}{2}}} \\ = \sum_{i = 1}^{m} {\frac{- 1}{2} [δ^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ (σ (y_{i}^{l})^{'} \cdot γ_{i}^{l})] (z_{i}^{l} - μ_{B}^{l}) [(D_{B}^{2})^{(l)} + ε]^{- \frac{3}{2}}} \\ (2.1) & = γ^{l} ⨀ \sum_{i = 1}^{m} {\frac{- 1}{2} [δ^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ σ (y_{i}^{l})^{'}] (z_{i}^{l} - μ_{B}^{l}) [(D_{B}^{2})^{(l)} + ε]^{- \frac{3}{2}}} \end{aligned}

$\begin{align} \frac {\partial C}{\partial (D^2_B)^l} &=\sum_{i=1}^{m}(\frac{\partial C}{\partial {\hat{z}^l_i}}\frac{\partial \hat{z}^l_i}{\partial (D^2_B)^l})\nonumber \\ &=\sum_{i=1}^{m}\lbrace\frac {-1} 2\frac{\partial C}{\partial {z}^{l+1}_i} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\frac{\partial a^l_i}{\partial y_i^l}\frac{\partial y_i^l}{\partial \hat{z}_i^l}{{(z^l_i-\mu^l_B)}}{[(D_B^2)^{(l)}+\varepsilon]^{-\frac 3 2}}\rbrace\nonumber \\ &=\sum_{i=1}^{m}\lbrace\frac {-1} 2[\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot(\sigma(y_i^l)'\cdot \gamma_i^l)]{{(z^l_i-\mu^l_B)}}{[(D_B^2)^{(l)}+\varepsilon]^{-\frac 3 2}}\rbrace\nonumber \\ &= \gamma^l\bigodot \sum_{i=1}^{m}\lbrace\frac {-1} 2[\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot \sigma(y_i^l)']{{(z^l_i-\mu^l_B)}}{[(D_B^2)^{(l)}+\varepsilon]^{-\frac 3 2}}\rbrace\nonumber \tag{2.1} \end{align}$

对均值求导:
这里需用到一个复合函数求导的方法:
这里写图片描述

所以按这个方法求导:

\begin{aligned} \frac{\partial C}{\partial μ_{B}^{l}} & = \sum_{i = 1}^{m} (\frac{\partial C}{\partial \hat{z_{i}}} \frac{\hat{z_{i}}}{\partial μ_{B}^{l}} + \frac{\partial C}{\partial D_{B}^{2}} \frac{\partial D_{B}^{2}}{\partial μ_{B}^{l}}) \\ = \sum_{i = 1}^{m} (\frac{\partial C}{\partial \hat{z_{i}}} \frac{- 1}{\sqrt{(D_{B}^{2})^{l} + ε}}) + \frac{\partial C}{\partial (D_{B}^{2})^{l}} \cdot \frac{- 2}{m} \cdot \sum_{i}^{m} (z_{i}^{l} - μ_{B}^{l}) \\ = \sum_{i = 1}^{m} (\frac{\partial C}{\partial \hat{z_{i}}} \frac{- 1}{\sqrt{(D_{B}^{2})^{l} + ε}}) \\ = \sum_{i = 1}^{m} (\frac{\partial C}{\partial z_{i}^{l + 1}} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} \frac{\partial a_{i}^{l}}{\partial y_{i}^{l}} \frac{\partial y_{i}^{l}}{\partial {\hat{z}}_{i}^{l}} \frac{- 1}{\sqrt{(D_{B}^{2})^{l} + ε}}) \\ (2.2) & = γ^{l} ⨀ \sum_{i = 1}^{m} ([δ^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ σ (y_{i}^{l})^{'}] \frac{- 1}{\sqrt{(D_{B}^{2})^{l} + ε}}) \end{aligned}

$\begin{align} {\frac{\partial C}{\partial \mu_B^l}}&=\sum_{i=1}^m(\frac {\partial C}{\partial \hat{z_i} }\frac { \hat{z_i}}{\partial \mu_B^l}+\frac{\partial C}{\partial D^2_B}\frac{\partial D^2_B}{\partial \mu_B^l})\nonumber \\ &=\sum_{i=1}^{m}(\frac {\partial C}{\partial \hat{z_i} }\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})+\frac{\partial C}{\partial (D^2_B)^l}\cdot \frac {-2} m \cdot \sum_i^m{(z^l_i-\mu^l_B)}\nonumber \\ &=\sum_{i=1}^{m}(\frac {\partial C}{\partial \hat{z_i} }\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})\nonumber \\ &=\sum_{i=1}^{m}(\frac{\partial C}{\partial {z}^{l+1}_i} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\frac{\partial a^l_i}{\partial y_i^l}\frac{\partial y_i^l}{\partial \hat{z}_i^l}\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})\nonumber \\ &=\gamma^l \bigodot \sum_{i=1}^{m}([\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot \sigma(y_i^l)']\frac{-1}{\sqrt{(D^2_B)^l+\varepsilon}})\tag{2.2} \end{align}$

\begin{aligned} δ_{i}^{l} = \frac{\partial C}{\partial z_{i}^{l}} & = \frac{\partial C}{\partial {\hat{z}}_{i}^{l}} \frac{\partial {\hat{z}}_{i}^{l}}{\partial z_{i}^{l}} + \frac{\partial C}{\partial D_{B}^{2}} \frac{\partial D_{B}^{2}}{\partial z_{i}^{l}} + \frac{\partial C}{\partial μ_{B}} \frac{\partial μ_{B}}{\partial z_{i}^{l}} \\ = \frac{\partial C}{\partial {\hat{z}}_{i}^{l}} \frac{1}{\sqrt{D_{B}^{2} + ε}} + \frac{\partial C}{\partial D_{B}^{2}} \cdot \frac{2}{m} \cdot (z_{i}^{l} - μ_{B}) + \frac{\partial C}{\partial μ_{B}} \cdot \frac{1}{m} \\ = \frac{\partial C}{\partial z_{i}^{l + 1}} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} \frac{\partial a_{i}^{l}}{\partial y_{i}^{l}} \frac{\partial y_{i}^{l}}{\partial {\hat{z}}_{i}^{l}} \frac{1}{\sqrt{D_{B}^{2} + ε}} + \frac{\partial C}{\partial D_{B}^{2}} \cdot \frac{2}{m} \cdot (z_{i}^{l} - μ_{B}) + \frac{\partial C}{\partial μ_{B}} \cdot \frac{1}{m} \\ (2.3) & = [δ^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ (σ (y_{i}^{l})^{'} ⨀ γ_{i}^{l})] \frac{1}{\sqrt{D_{B}^{2} + ε}} + \frac{\partial C}{\partial D_{B}^{2}} \cdot \frac{2}{m} \cdot (z_{i}^{l} - μ_{B}) + \frac{\partial C}{\partial μ_{B}} \cdot \frac{1}{m} \end{aligned}

$\begin{align} \delta^l_i=\frac{\partial C}{\partial z^l_i} &=\frac{\partial C}{\partial \hat{z}^l_i}\frac{\partial \hat{z}^l_i}{\partial z^l_i}+\frac{\partial C}{\partial D^2_B}\frac{\partial D^2_B}{\partial z^l_i}+\frac{\partial C}{\partial \mu_B}\frac {\partial \mu_B}{\partial z^l_i}\nonumber\\ &=\frac{\partial C}{\partial \hat{z}^l_i}\frac{1}{\sqrt{D^2_B+\varepsilon}}+\frac{\partial C}{\partial D^2_B}\cdot \frac 2 {m} \cdot (z^l_i-\mu_B)+\frac{\partial C}{\partial \mu_B}\cdot \frac 1 m\nonumber\\ &=\frac{\partial C}{\partial {z}^{l+1}_i} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\frac{\partial a^l_i}{\partial y_i^l}\frac{\partial y_i^l}{\partial \hat{z}_i^l}\frac{1}{\sqrt{D^2_B+\varepsilon}}+\frac{\partial C}{\partial D^2_B}\cdot \frac 2 {m} \cdot (z^l_i-\mu_B)+\frac{\partial C}{\partial \mu_B}\cdot \frac 1 m\nonumber\\ &=[\delta^{l+1} \frac{\partial {z}^{l+1}_i}{\partial a^l_i}\bigodot(\sigma(y_i^l)'\bigodot \gamma_i^l)]\frac{1}{\sqrt{D^2_B+\varepsilon}}+\frac{\partial C}{\partial D^2_B}\cdot \frac 2 {m} \cdot (z^l_i-\mu_B)+\frac{\partial C}{\partial \mu_B}\cdot \frac 1 m \tag{2.3} \end{align}$

则，以全连接层为例，求权重和偏差梯度：

\begin{matrix} (2.4) & \frac{\partial C}{\partial w^{l}} = \sum_{i = 1}^{m} \frac{\partial C}{\partial z_{i}^{l}} \frac{\partial z_{i}^{l}}{\partial w^{l}} = \sum_{i = 1}^{m} (a^{l - 1})^{T} δ^{l} \end{matrix}

$\frac {\partial C}{\partial w^l}=\sum_{i=1}^{m}\frac{\partial C}{\partial z_i^l}\frac{\partial z_i^l}{\partial w^l}=\sum_{i=1}^m(a^{l-1})^T\delta^l\tag{2.4}$

\begin{matrix} (2.5) & \frac{\partial C}{\partial β^{l}} = \sum_{i = 1}^{m} \frac{\partial C}{\partial z_{i}^{l + 1}} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} \frac{\partial a_{i}^{l}}{\partial y_{i}^{l}} \frac{\partial y_{i}^{l}}{\partial β^{l}} = \sum_{i = 1}^{m} δ_{i}^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ [σ (y_{i}^{l})^{'}) \frac{\partial y^{l}}{\partial β^{l}}] = \sum_{i = 1}^{m} δ_{i}^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ σ (y_{i}^{l})^{'} \end{matrix}

$\frac{\partial C}{\partial \beta^l}=\sum_{i=1}^{m}\frac{\partial C}{\partial {z}_i^{l+1}}\frac{\partial {z}_i^{l+1}}{\partial a_i^l}\frac{\partial a_i^l}{\partial {y}_i^l}\frac {\partial {y}_i^l}{\partial \beta^l}=\sum_{i=1}^{m}\delta_i^{l+1} \frac{\partial {z}_i^{l+1}}{\partial a_i^l}\bigodot [\sigma(y_i^l)')\frac {\partial {y}^l}{\partial \beta^l}]=\sum_{i=1}^{m}\delta_i^{l+1} \frac{\partial {z}_i^{l+1}}{\partial a_i^l}\bigodot \sigma(y_i^l)'\tag{2.5}$

同理:

\begin{matrix} (2.6) & \frac{\partial C}{\partial γ^{l}} = \sum_{i = 1}^{m} \frac{\partial C}{\partial z_{i}^{l + 1}} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} \frac{\partial a_{i}^{l}}{\partial y_{i}^{l}} \frac{\partial y_{i}^{l}}{\partial γ^{l}} = \sum_{i = 1}^{m} δ_{i}^{l + 1} \frac{\partial z_{i}^{l + 1}}{\partial a_{i}^{l}} ⨀ [σ (y_{i}^{l})^{'} ⨀ {\hat{z}}_{i}^{l}] \end{matrix}

$\frac {\partial C}{\partial \gamma^l}=\sum_{i=1}^{m}\frac{\partial C}{\partial {z}_i^{l+1}}\frac{\partial {z}_i^{l+1}}{\partial a_i^l}\frac{\partial a_i^l}{\partial {y}_i^l}\frac {\partial {y}_i^l}{\partial \gamma^l}=\sum_{i=1}^{m}\delta_i^{l+1} \frac{\partial {z}_i^{l+1}}{\partial a_i^l}\bigodot[ \sigma(y_i^l)'\bigodot \hat{z}_i^l]\tag{2.6}$

对应yolo代码:

void backward_batchnorm_layer(layer l, network net)
{
    //非训练状态
    if(!net.train){
        l.mean = l.rolling_mean;
        l.variance = l.rolling_variance;
    }
    //求偏差beta的梯度,对应公式2.5
    backward_bias(l.bias_updates, l.delta, l.batch, l.out_c, l.out_w*l.out_h);

    //求gamma梯度,对应公式2.6
    backward_scale_cpu(l.x_norm, l.delta, l.batch, l.out_c, l.out_w*l.out_h, l.scale_updates);


     //先计算公式2.3里面的公共项gamma
    scale_bias(l.delta, l.scales, l.batch, l.out_c, l.out_h*l.out_w);

    //求y对均值的导数,对应公式2.2
    mean_delta_cpu(l.delta, l.variance, l.batch, l.out_c, l.out_w*l.out_h, l.mean_delta);

    //求y对方差的导数,对应公式2.1,这里按上面化简后的公式,若激活函数为relu应该直接等于0
    variance_delta_cpu(l.x, l.delta, l.mean, l.variance, l.batch, l.out_c, l.out_w*l.out_h, l.variance_delta);

    //求权重的误差度,对应公式2.3
    normalize_delta_cpu(l.x, l.mean, l.variance, l.mean_delta, l.variance_delta, l.batch, l.out_c, l.out_w*l.out_h, l.delta);

     //对于BATCHNORM层,直接输出等于输入
    if(l.type == BATCHNORM) copy_cpu(l.outputs*l.batch, l.delta, 1, net.delta, 1);
}

4.预测：

预测时，计算总体的均值和方差是不实际的，也是无法实现的，因为无法采样到所有样本。用总采样来估计总体的均值和方差呢？也是需要大量计算的，在训练过程中的batch下的均值uB和方差σB，可以加以利用来估计总体
具体推导如下:

$E(X_i).D(X_i)$ :可以理解为 $X_i$ 所在分布的期望值

样本均值:

\begin{matrix} (3.1) & μ_{A} = \bar{X} = \frac{1}{m} \sum_{i = 1}^{m} X_{i} \end{matrix}

$\mu_A=\bar X=\frac 1 m \sum_{i=1}^mX_i \tag{3.1}$
因为抽样和样本同分布,所以:
样本期望:

\begin{matrix} (3.2) & μ = E (X_{i}) = E (μ_{A}) = E (x) \end{matrix}

$\mu=E(X_i)=E(\mu_A)=E(x)\tag{3.2}$
但

\begin{matrix} (3.3) & μ_{A} = E (X_{A_{i}}) \neq E (X_{i}) \end{matrix}

$\mu_A=E(X_{A_i})\neq E(X_i)\tag{3.3}$
样本方差:

\begin{matrix} (3.4) & σ^{2} = D (X_{i}) = D (x) \end{matrix}

$\sigma^2=D(X_i)=D(x)\tag{3.4}$

所 以 :

$所以:$

\begin{matrix} (3.5) & μ = \frac{1}{n} \sum_{i = 1}^{n} E (X_{i}) \end{matrix}

$\mu=\frac 1 n\sum_{i=1}^nE(X_i)\tag{3.5}$ $

均值和方差:

\begin{matrix} (3.6) & E (μ_{A}) = E (\frac{1}{m} \sum_{i = 1}^{m} X_{i}) = \frac{1}{z} m n \sum_{i = 1}^{m n} X_{i} = E (x) \end{matrix}

$E(\mu_A)=E(\frac 1 m \sum_{i=1}^mX_i)=\frac 1 zmn \sum_{i=1}^{mn}X_i=E(x)\tag{3.6}$

\begin{matrix} (3.7) & D (μ_{A}) = D (\frac{1}{m} \sum_{i = 1}^{m} X_{i}) = \frac{1}{m^{2}} \sum_{i = 1}^{m} D (X_{i}) = \frac{1}{m} D (x) \end{matrix}

$D(\mu_A)=D(\frac 1 m\sum_{i=1}^mX_i)=\frac 1 {m^2}\sum_{i=1}^mD(X_i)=\frac 1 m D(x)\tag{3.7}$

\begin{aligned} E (σ_{A}^{2}) & = E [\sum_{i = 1}^{m} (X_{i} - μ_{A})^{2} \cdot \frac{1}{m}] \\ = \frac{1}{m} \cdot E {\sum_{i = 1}^{m} [X_{i} - E (x) + E (x) - μ_{A}]^{2}} \\ = \frac{1}{m} \cdot E {\sum_{i = 1}^{m} [(X_{i} - E (x))^{2} - 2 (X_{i} - E (x) (μ_{A} - E (x) + (μ_{A} - E (x))^{2}]} \\ = \frac{1}{m} \cdot E {\sum_{i = 1}^{m} [(X_{i} - E (x))^{2}] - 2 m (μ_{A} - E (x))^{2} + m (μ_{A} - E (x))^{2}} \\ = \frac{1}{m} \cdot E {\sum_{i = 1}^{m} [(X_{i} - E (x))^{2}] - m (μ_{A} - E (x))^{2}} \\ = D (x) - E {[μ_{A} - E (μ_{A})]^{2}} \\ = D (x) - D (μ_{A}) \\ (1) & = \frac{m - 1}{m} D (x) \end{aligned}

$\begin{align} E(\sigma^2_A) &=E[\sum_{i=1}^m(X_i-\mu_A)^2\cdot \frac 1 m] \nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[X_i-E(x)+E(x)-\mu_A]^2\rbrace\nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[(X_i-E(x))^2-2(X_i-E(x)(\mu_A-E(x)+(\mu_A-E(x))^2]\rbrace\nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[(X_i-E(x))^2]-2m(\mu_A-E(x))^2+m(\mu_A-E(x))^2\rbrace\nonumber \\ &=\frac 1 m \cdot E\lbrace\sum_{i=1}^m[(X_i-E(x))^2]-m(\mu_A-E(x))^2\rbrace\nonumber \\ &=\ D(x)- E\lbrace[\mu_A-E(\mu_A)]^2\rbrace\nonumber \\ &=D(x)-D(\mu_A)\nonumber \\ &=\frac {m-1} mD(x)\end{align}\nonumber \\$

即: $D(x)=\frac m {m-1}E(\sigma^2_A)\tag{3.8}$

所以最终结果:
$\mu =E(x)=E(\mu_B)=\frac 1 K \sum_B^K \mu_B\tag{K为总迭代次数}$
$D(x)=\frac m {m-1}E(\sigma^2_A)\tag{3.8}$ tag{m为batch_size}$

虽然理论上是这样的,但yolo里面好像用滚动平均值来算,而不是上面的计算方法这部分计算对应前行传播函数forward_batchnorm_layer()里的:

scal_cpu(l.out_c, .99, l.rolling_mean, 1);
axpy_cpu(l.out_c, .01, l.mean, 1, l.rolling_mean, 1);
scal_cpu(l.out_c, .99, l.rolling_variance, 1);
axpy_cpu(l.out_c, .01, l.variance, 1, l.rolling_variance, 1);

Bacthnormal优化办法

1)增大学习率.
BN能减少每层的梯度变化幅度,使梯度稳定在理想的变化范围内,所以大学习率一般不会导致梯度消失
另外大学习率,训练一次,一般会导致参数,如权重变大,假设变大了n倍,即 $W'=nW$ ,其中W’为目前权重,W’为上次训练的权重

设上一次训练,均值为:
$\mu_B=\frac 1 m\sum_{i=1}^mWa^{l-1}$
方差:
$D_B^2=\frac 1 m\sum_{i=1}^m(Wa^{l-1}-\mu_B)^2$

则本次均值:
$\mu_B'=\frac 1 m\sum_{i=1}^mnWa^{l-1}=n\mu_B$
$(D_B^2)'=\frac 1 m\sum_{i=1}^m(nWa^{l-1}-\mu_B')^2=n^2D_B^2$

所以:
$BN(Wa^{l-1})=BN(nWa^{l-1})$
$\frac{\partial BN(nWa^{l-1})}{\partial (nW)}=\frac 1 n\frac{\partial BN(Wa^{l-1})}{\partial W}$
即:
$\frac{\partial BN(W'a^{l-1})}{\partial (W')}=\frac 1 n\frac{\partial BN(Wa^{l-1})}{\partial W}$
可以看出,当因上一次训练的大学习率导致权重W变大n倍后,只会让本次训练的梯度更小,这样一点程度上,便避免了过大学习率导致的梯度爆炸.

2)去掉Dropout_
3)减少L2正则项_,这里不是很理解,难道两者冲突了?
4)提高学习率衰减速度.由于BN收敛速度快,在相同迭代次数,使用BN算法的网络,比未使用相同BN算法的网络,会更快到达相应的衰减点
5)更彻底随机化训练数据._
6)减少图片扭曲_

看了好久,终于略懂一二 :D
主要参考:
论文
 Batch Normalize的几点说明
 https://zhuanlan.zhihu.com/p/27938792