本文主要是BP网络的前后向传播较详细推导，以及C++实现，记下来也方便后面的回顾，也希望对关系细节的读者也一丝帮助。如果有不对的地方，请指正。

BP图模型：

网络中单个激活单元：

• 上图定了隐层中的激活单元，该隐层激活单元中含有一个偏置项b。相关运算如图所示，符号右上角角标为单元在网络中的层好，结合代码实现时，网络激活单元之间的权重一般保存在前一层的单元中。
  

这里有两点注意：

1. 输入层的单元中没有偏置项b，但有权值项w。
2. 输出层的单元中没有权值项w，但有偏置项b。

相关符号定义：

符号	意义
m	样本数
$nl$	网络的总层数
${{\rm{L}}_l}$	第$l$层
${\rm{W}}_{ij}^{(l)}$	$l$层的j单元与$l+1$的i单元之间的权值（weight）
${\rm{b}}_i^{(l)}$	第$l$层第i个单元的偏置项
${{\rm{S}}_l}$	表示第$l$ 层的节点数
$a_i^{(l)}$	表示第$l$层第i单元的激活值
f(x)	sigmoid函数：${\rm{f}}(x) = \frac{1}{{1 + {e^{ - x}}}}$
$z_i^{(l)}$	表示第$l$层第i单元的输入
${{h_{wb}}(x)}$	表示整个网络对输入x的输出结果，等价于$a^{({\rm{n}}l)}$

损失函数（带2范式正则）：

$$J(W,b) = \frac{1}{m}\sum\limits_{{\rm{i = 1}}}^{\rm{m}} {J(W,b;{x^i},{y^i})} {\rm{ + }}\frac{\lambda }{2}{\sum\limits_{l = 1}^{nl - 1} {\sum\limits_{i = 1}^{{S_l}} {\sum\limits_{j = 1}^{{S_l} + 1} {\left( {W_{ji}^{(l)}} \right)} } } ^2}$$
其中， $${\rm{J}}(W,b;x,y) = \frac{1}{2}{\left\| {{h_{wb}}(x) - y} \right\|^2}$$

我们优化所有权值和偏置就是通过最小化损失函数来实现的，通过对损失函数计算各权值和偏置的梯度，然后沿着各自梯度的反方向走，就可以让损失函数慢慢变小，由于神经网络不是的损失函数不是严格凸函数，所以并不能保证找到全局最优解。我们首先就要计算各权值和偏置关于损失函数的梯度。

前向传播

Tip : 这里需要先初始化各单元中的权值和偏置项的值，权值可以按照标准正态分布去产生，也可以使用其它方式产生，但最好不要偷懒而给所有权值赋上相同的值，这样会导致极慢的收敛速度，有兴趣的读者可以修改下面的程序自己试下。

1. 输入层向隐层的前向传播：
   
   • 向输入层输入数据X，第二层第i个激活单元相关计算：
     该激活单元的输入：$z_i^{(2)}{\rm{ = }}\sum\limits_{{\rm{j}} = 1}^{{S_1}} {W_{ij}^{(1)}a_j^{(1)} + b_i^{(2)}}$，其中$a_i^{(1)}{\rm{ = }}{{\rm{X}}_i}$
     该激活单元的输入：$a_i^{(2)}{\rm{ = f}}(z_i^{(2)}) = \frac{1}{{1 + {e^{ - z_i^{(2)}}}}}$
     计算完输入层的前向传播后就可以计算隐层间的传播了。
2. 隐层间的传播：
   
   • 上一层的隐层输出值作为当前隐层的输入值：
     第l 层第i个激活单元输入：$z_i^{(l)}{\rm{ = }}\sum\limits_{{\rm{j}} = 1}^{{S_{l-1}}} {W_{ij}^{(l-1)}a_j^{(l-1)} + b_i^{(l)}}$
     第l 层第i个激活单元输出：$a_i^{(l)}{\rm{ = f}}(z_i^{(l)}) = \frac{1}{{1 + {e^{ - z_i^{(l)}}}}}$
     隐层按照从低向高的顺序依次计算各层的激活单元，依次计算各层：$l=3,4,...{{S_{nl-1}}}$
3. 输出层的传播：
   
   • 传播到最后的输出层：
     第${{S_{nl}}}$层第i个激活单元输入：$z_i^{(nl)}{\rm{ = }}\sum\limits_{{\rm{j}} = 1}^{{S_{nl-1}}} {W_{ij}^{(nl-1)}a_j^{(nl-1)} + b_i^{(nl)}}$
     第${{S_{nl}}}$层第i个激活单元最终的输出：$a_i^{(nl)}{\rm{ = f}}(z_i^{(nl)}) = \frac{1}{{1 + {e^{ - z_i^{(nl)}}}}}$
     其中$a_i^{(nl)}$就是神经网络最终的输出。

反向传播

Tip : 如果直接对损失函数中位于第一层的权值求导，会发现无从下手，因为后面的所有层的激活单元都直接或间接的包含了第一层的权值，这是一种嵌套的关系（数学函数嵌套，类似于斐波那契数列），第一层的权值被嵌套的最深。换言之，约靠后的激活单元，其权值在损失函数中嵌套的就越浅。既然这样，那我们可以先从靠后的权值下手，比如倒数第二层的权值${W^{nl-1}}$，在损失函数中，就是最顶层的（输出层没有权值）。为了后期计算的方便，我们从后向前对各层各激活单元的输入变量${Z_i^{l}}$求导。后在对${W_i^{l}}$求导就显得非常简单了，这里主要是求导的链式法则起到了关键作用（题外话：RNN中的LSTM单元也是通过这种思想求解，可以将时间t比作层数来从后向前计算）。

1. 反向传播之输出层：
   $$\begin{array}{l} \delta _i^{nl}{\rm{ = }}\frac{{\partial {\rm{J}}(W,b;x,y)}}{{\partial Z_i^{nl}}}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{{\partial \frac{1}{2}{{\left\| {{h_{wb}}(x) - y} \right\|}^2}}}{{\partial Z_i^{nl}}} = \frac{{\partial \frac{1}{2}{{\left\| {a_i^{nl} - y} \right\|}^2}}}{{\partial Z_i^{nl}}}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{{\partial \frac{1 }{2}{{\left\| {f(Z_i^{nl}) - y} \right\|}^2}}}{{\partial Z_i^{nl}}}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \left( {a_i^{nl} - y_i} \right)\frac{{\partial f(Z_i^{nl})}}{{\partial Z_i^{nl}}}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \left( {a_i^{nl} - y_i} \right)f(Z_i^{nl})\left( {1 - f(Z_i^{nl})} \right) \end{array}$$
2. 反向传播之隐藏层和输入层：
   这里假设我们已经将后一层的${\delta _i^{l+1}}$计算出来了（实际上输出层（最后一层）上的${\delta _i^{nl}}$已经根据上面的公式计算出来了），下面计算来计算${\delta _i^{l}}$：

$$\begin{array}{l} \delta _i^l{\rm{ = }}\frac{{\partial {\rm{J}}(W,b;x,y)}}{{\partial Z_i^l}}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\rm{ = }}\frac{{\partial \left( {\frac{1}{2}\sum\limits_{j = 1}^{{S_{nl}}} {{{\left( {a_j^{nl} - {y_j}} \right)}^2}} } \right)}}{{\partial Z_i^l}} = \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\frac{{\partial \left( {\frac{1}{2}\sum\limits_{j = 1}^{{S_{nl}}} {{{\left( {a_j^{nl} - {y_j}} \right)}^2}} } \right)}}{{\partial Z_k^{l + 1}}} \times \frac{{\partial Z_k^{l + 1}}}{{\partial Z_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\delta _k^{l + 1} \times \frac{{\partial Z_k^{l + 1}}}{{\partial Z_i^l}}} \right)} = \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\delta _k^{l + 1} \times \frac{{\partial \left( {\sum\limits_{t = 1}^{{S_l}} {\left( {W_{kt}^la_t^l} \right)+b_k^{l+1}} } \right)}}{{\partial Z_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\delta _k^{l + 1} \times \frac{{\partial \left( {\sum\limits_{t = 1}^{{S_l}} {\left( {W_{kt}^lf(Z_t^l)} \right)+b_k^{l+1}} } \right)}}{{\partial Z_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\delta _k^{l + 1} \times W_{ki}^l \times \frac{{\partial f(Z_i^l)}}{{\partial Z_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\delta _k^{l + 1} \times W_{ki}^l \times f(Z_i^l) \times (1 - f(Z_i^l))} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = f(Z_i^l) \times (1 - f(Z_i^l)) \times \sum\limits_{k = 1}^{{S_{l + 1}}} {\left( {\delta _k^{l + 1} \times W_{ki}^l} \right)} \end{array}$$

计算各权值和偏置的梯度：

更新策略：对于每一个样本${(x^i,y^i)}$，先使用前向传播计算出各激活单元的输出值${a_i^l}$，再通过反向传播，计算出各激活单元的输入关于单个样本对应损失函数的偏导${\delta_i^l}$，之后就可以是用下面的公式对所有样本求出每个权值和偏置的偏导，并使用优化算法对其更新，这里使用的是梯度下降，也可以使用SGD等优化算法。

• 权值关于损失函数的梯度：
  

  $$\begin{array}{l} \frac{{\partial {\rm{J}}(W,b)}}{{\partial W_{ji}^{(l)}}} = \frac{1}{m}\sum\limits_{t = 1}^m {\frac{{\partial J(W,b;{x^t},{y^t})}}{{\partial W_{ji}^{(l)}}}} + \lambda W_{ji}^{(l)}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\frac{{\partial J(W,b;{x^t},{y^t})}}{{\partial Z_j^{l + 1}}} \times \frac{{\partial Z_j^{l + 1}}}{{\partial W_{ji}^{(l)}}}} \right)} + \lambda W_{ji}^{(l)}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\delta _j^{l + 1} \times \frac{{\partial Z_j^{l + 1}}}{{\partial W_{ji}^{(l)}}}} \right)} + \lambda W_{ji}^{(l)}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\delta _j^{l + 1} \times \frac{{\partial \left( {\sum\limits_{k = 1}^{{S_l}} {\left( {W_{jk}^la_k^l} \right) + b_i^l} } \right)}}{{\partial W_{ji}^{(l)}}}} \right)} + \lambda W_{ji}^{(l)}\\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\delta _j^{l + 1} \times a_i^l} \right)} + \lambda W_{ji}^{(l)} \end{array}$$

• 偏置关于损失函数的梯度：

$$\begin{array}{l} \frac{{\partial {\rm{J}}(W,b)}}{{\partial b_i^l}} = \frac{1}{m}\sum\limits_{t = 1}^m {\frac{{\partial J(W,b;{x^t},{y^t})}}{{\partial b_i^l}}} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\frac{{\partial J(W,b;{x^t},{y^t})}}{{\partial Z_j^{l + 1}}} \times \frac{{\partial Z_j^{l + 1}}}{{\partial b_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\delta _j^{l + 1} \times \frac{{\partial Z_j^{l + 1}}}{{\partial b_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\delta _j^{l + 1} \times \frac{{\partial \left( {\sum\limits_{k = 1}^{{S_l}} {\left( {W_{jk}^la_k^l} \right) + b_i^l} } \right)}}{{\partial b_i^l}}} \right)} \\ {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} {\kern 1pt} = \frac{1}{m}\sum\limits_{t = 1}^m {\left( {\delta _j^{l + 1}} \right)} \end{array}$$

程序中相关变量解释：

变量名	代表含义
value	对应单元的激活值：$a_i^{(l)}$
rightout	样本真实值：${{y^i}}$
error	损失误差：$\frac{1}{m}\sum\limits_{{\rm{i = 1}}}^{\rm{m}} {J(W,b;{x^i},{y^i})}$
delta	单个样本偏差：${\frac{{\partial J(W,b;{x^i},{y^i})}}{{\partial W_{ji}^{(l)}}}}$
wDeltaSum	整个样本偏差和，即最终的权值偏差： $\frac{{\partial {\rm{J}}(W,b)}}{{\partial W_{ji}^{(l)}}}$

---

下面是c++版实现程序（没有使用正则）：

• 注：程序实现的是对XOR运算的拟合

BpNet.h：

#include<iostream>
#include<cmath>
#include<vector>
#include<stdlib.h>
#include<time.h>

using namespace std;

#define innode 2            //输入节点数
#define hidenode 4          //隐含节点数
#define hidelayer 1         //隐含层数
#define outnode 1           //输出节点数
#define learningRate 0.9    //学习速率，alpha

//产生随机数
inline double get_11Random()
{
    return ((2.0*(double)rand()/RAND_MAX) - 1);
}

//sigmoid函数
inline double sigmoid(double x)
{
    double ans = 1.0/(1+exp(-x));
    return ans;
}

/*输入层节点
//1.value:      固定输入值
//2.weight:     面对第一层隐含层每个节点都有权值
//3.wDeltaSum:  面对第一层隐含层每个节点权值的delta值积累
*/
typedef struct inputNode
{
    double value;
    vector<double> weight,wDeltaSum;
}inputNode;

/*输出层节点
  1.value:      节点当前值
  2.delta:      与正确输出值之间的delta值
  3.rightout:   正确输出值
  4.bias:       偏移量
  5.bDeltaSum:  bias的delta值的积累，每个节点一个
 */
typedef struct outputNode
{
    double value, delta, rightout, bias, bDeltaSum;
}outputNode;

/*隐含层节点
  1.value:      节点当前值
  2.delta:      BP推导出的delta值
  3.bias:       偏移量
  4.bDeltaSum:  bias的delta值的积累，每个节点一个
  5.weight:     面对下一层每个节点都有的权值
  6.wDeltaSum:  weight的delta值的积累，面对下一层每个节点各自积累
 */
typedef struct hiddenNode
{
    double value, delta, bias, bDeltaSum;
    vector<double> weight, wDeltaSum;
}hiddenNode;

//单个样本
typedef struct sample
{
    vector<double> in, out;
}sample;

//BP神经网络
class BpNet
{
    public:
        BpNet();        //构造函数
        void forwardPropagationEpoc();      //单个样本前向传播
        void backPropagationEpoc();         //单个样本后向传播
        void training (static vector<sample> sampleGroup, double threshold);    //更新weight,bias
        void predict(vector<sample>& testGroup);            //神经网络预测
        void setInput(static vector<double> sampleIn);      //设置学习样本输入
        void setOutput(static vector<double> sampleOut);    //设置学习样本输出
        double error;
        inputNode* inputLayer[innode];                  //输入层
        outputNode* outputLayer[outnode];               //输出层
        hiddenNode* hiddenLayer[hidelayer][hidenode];   //隐含层
};

BpNet.cpp:

#include "BpNet.h"
using namespace std;

BpNet::BpNet()
{
    srand((unsigned)time(NULL));
    error = 100.f;

    //初始化输入层
    for(int i = 0; i< innode; i++)
    {
        inputLayer[i] = new inputNode();
        for(int j = 0; j < hidenode; j++)
        {
            inputLayer[i]->weight.push_back(get_11Random());
            inputLayer[i]->wDeltaSum.push_back(0.f);
        }
    }

    //初始化隐藏层
    for ( int i = 0; i < hidelayer; i++)
    {
        if ( i ==hidelayer - 1)
        {
            for(int j = 0;j < hidenode; j++ )
            {
                hiddenLayer[i][j] = new hiddenNode();
                hiddenLayer[i][j]->bias = get_11Random();
                for (int k = 0;k < outnode; k++)
                {
                    hiddenLayer[i][j]->weight.push_back(get_11Random());
                    hiddenLayer[i][j]->wDeltaSum.push_back(0.f);
                }
            }
        }
        else
        {
            for (int j =0; j < hidenode; j++)
            {
                hiddenLayer[i][j] = new hiddenNode();
                hiddenLayer[i][j]->bias = get_11Random();
                for (int k = 0; k < hidenode; k++)
                {
                    hiddenLayer[i][j]->weight.push_back(get_11Random());
                    hiddenLayer[i][j]->wDeltaSum.push_back(0.f);
                }
            }
        }

    }

    //初始化输出层
    for ( int i = 0; i < outnode; i++)
    {
        outputLayer[i] = new outputNode();
        outputLayer[i]->bias = get_11Random();
    }
}

void BpNet::forwardPropagationEpoc()
{
    //forward propagation on hidden layer
    for ( int i = 0; i < hidelayer; i++)
    {
        if (i == 0 )
        {
            for ( int j = 0; j < hidenode; j++)
            {
                double sum = 0.f;
                for ( int k = 0; k < innode; k++)
                {
                    sum += inputLayer[k]->value * inputLayer[k]->weight[j];
                }
                sum += hiddenLayer[i][j]->bias;
                hiddenLayer[i][j]->value = sigmoid(sum);
            }
        }
        else
        {
            for ( int j = 0; j < hidenode; j++)
            {
                double sum = 0.f;
                for ( int k = 0; k < hidenode; k++)
                {
                    sum += hiddenLayer[i-1][k]->value*hiddenLayer[i-1][k]->weight[j];
                }
                sum += hiddenLayer[i][j]->bias;
                hiddenLayer[i][j]->value = sigmoid(sum);
            }
        }
    }

    //forward propagation on output layer
    for ( int i = 0; i < outnode; i++)
    {
        double sum = 0.f;
        for( int j = 0; j < hidenode; j++)
        {
            sum += hiddenLayer[hidelayer - 1][j]->value * hiddenLayer[hidelayer - 1][j]->weight[i];
        }
        sum += outputLayer[i]->bias;
        outputLayer[i]->value = sigmoid(sum);
    }

}

void BpNet::backPropagationEpoc()
{
    // backward propagation on output layer
    // -- comput delta

    for ( int i = 0; i < outnode; i++)
    {
        double temp = fabs(outputLayer[i]->value-outputLayer[i]->rightout);
        error += temp * temp / 2;
        outputLayer[i]->delta = (outputLayer[i]->value - outputLayer[i]->rightout)*(1-outputLayer[i]->value)*outputLayer[i]->value;

    }

    // backward propagation on hidden layer
    // compute delta
    for ( int i = hidelayer - 1; i >= 0; i--)
    {
        if ( i == hidelayer - 1)
        {
            for ( int j = 0; j < hidenode; j++)
            {
                double sum = 0.f;
                for ( int k = 0; k < outnode; k++)
                {
                    sum += outputLayer[k]->delta * hiddenLayer[i][j]->weight[k];
                }
                hiddenLayer[i][j]->delta = sum*hiddenLayer[i][j]->value*(1 - hiddenLayer[i][j]->value);
            }
        }
        else
        {
            for ( int j = 0; j < hidenode; j++)
            {
                double sum = 0.f;
                for ( int k = 0; k < hidenode; k++)
                {
                    sum += hiddenLayer[i + 1][k]->delta * hiddenLayer[i][j]->weight[k];
                }
                hiddenLayer[i][j]->delta = sum * hiddenLayer[i][j]->value*(1-hiddenLayer[i][j]->value);
            }
        }

    }

    // backward propagation on input layer
    // update weight delta sum
    for ( int i = 0; i < innode; i++)
    {
        for ( int j = 0; j < hidenode; j++)
        {
            inputLayer[i]->wDeltaSum[j] += inputLayer[i]->value*hiddenLayer[0][j]->delta;

        }
    }

    // backward propagation on hidden layer
    // update weight delta sum and bias delta sum
    // 计算偏导数
    for ( int i = 0; i < hidelayer; i++)
    {
        if ( i == hidelayer - 1)
        {
            for ( int j = 0; j < hidenode; j++)
            {
                hiddenLayer[i][j]->bDeltaSum += hiddenLayer[i][j]->delta;
                for ( int k = 0; k < outnode; k++)
                {
                    hiddenLayer[i][j]->wDeltaSum[k] += hiddenLayer[i][j]->value * outputLayer[k]->delta;
                }
            }
        }
        else
        {
            for (int j = 0; j < hidenode; j++)
            {
                hiddenLayer[i][j]->bDeltaSum += hiddenLayer[i][j]->delta;
                for ( int k = 0; k < hidenode; k++)
                {
                    hiddenLayer[i][j]->wDeltaSum[k] += hiddenLayer[i][j]->value * hiddenLayer[i+1][k]->delta;
                }
            }
        }

    }

    // backward propagation on output layer
    // update bias delta sum
    for ( int i = 0; i < outnode; i++)
    {
        outputLayer[i]->bDeltaSum += outputLayer[i]->delta;
    }

}

void BpNet::training(static vector<sample> sampleGroup, double threshold)
{
    int sampleNum = sampleGroup.size();

    while ( error > threshold)
    {
        cout << "training error:"<<error<<endl;
        error = 0.f;
        //initialize delta sum
        for ( int i = 0; i < innode; i++)
        {
            inputLayer[i]->wDeltaSum.assign(inputLayer[i]->wDeltaSum.size(), 0.f);
        }
        for ( int i = 0; i < hidelayer; i++)
        {
            for ( int j = 0; j < hidenode; j++)
            {
                hiddenLayer[i][j]->wDeltaSum.assign(hiddenLayer[i][j]->wDeltaSum.size(), 0.f);
                hiddenLayer[i][j]->bDeltaSum = 0.f;
            }

        }
        for ( int i = 0; i < outnode; i++)
        {
            outputLayer[i]->bDeltaSum = 0.f;
        }

        // start training
        for( int iter = 0; iter < sampleNum; iter++)
        {
            // initial data of input and output
            setInput(sampleGroup[iter].in);
            setOutput(sampleGroup[iter].out);
            // forward and backward propagation
            // compute delta
            forwardPropagationEpoc();
            backPropagationEpoc();
        }

        // deltasum had computed over! then update weight and bias

        // backward propagation on input layer
        // update weight with Gradient descent
        for ( int i = 0; i < innode; i++)
        {
            for ( int j = 0; j < hidenode; j++)
            {
                inputLayer[i]->weight[j] -= learningRate * inputLayer[i]->wDeltaSum[j] /sampleNum;
            }
        }

        // backward propagation on hidden layer
        // update weight and bias
        for ( int i = 0; i < hidelayer; i++)
        {
            if( i == hidelayer -1)
            {
                for ( int j = 0;  j < hidenode; j++)
                {
                    // update bias
                    hiddenLayer[i][j]->bias -= learningRate * hiddenLayer[i][j]->bDeltaSum/sampleNum;
                    // update weight
                    for ( int k = 0; k < outnode; k++)
                    {
                        hiddenLayer[i][j]->weight[k] -= learningRate * hiddenLayer[i][j]->wDeltaSum[k]/sampleNum;
                    }
                }
            }
            else
            {
                for ( int j = 0; j < hidenode; j++)
                {
                    // update bias
                    hiddenLayer[i][j]->bias -= learningRate * hiddenLayer[i][j]->bDeltaSum/sampleNum;
                    // update weight
                    for ( int k = 0; k < hidenode; k++)
                    {
                        hiddenLayer[i][j]->weight[k] -= learningRate * hiddenLayer[i][j]->wDeltaSum[k]/sampleNum;
                    }
                }
            }
        }

        // backward propagation on output layer
        // udate bias
        for ( int i = 0; i < outnode; i++)
        {
            outputLayer[i]->bias -= learningRate * outputLayer[i]->bDeltaSum/sampleNum;
        }

    }

}

void BpNet::predict(vector<sample>& testGroup)
{
     int testNum = testGroup.size();
     for(int iter = 0; iter < testNum; iter++)
     {
         testGroup[iter].out.clear();
         setInput(testGroup[iter].in);
         forwardPropagationEpoc();
         for ( int i = 0; i< outnode; i++)
         {
             testGroup[iter].out.push_back(outputLayer[i]->value);
         }

     }

}

void BpNet::setInput(static vector<double> sampleIn)
{
    for ( int i = 0; i < innode; i++)
    {
        inputLayer[i]->value = sampleIn[i];
    }
}

void BpNet::setOutput(static vector<double> sampleOut)
{
    for ( int i = 0; i < outnode; i++)
    {
        outputLayer[i]->rightout = sampleOut[i];
    }
}


int main()
{
    BpNet testNet;

    //学习样本
    vector<double> samplein[4];
    vector<double> sampleout[4];

    samplein[0].push_back(0);
    samplein[0].push_back(0);
    sampleout[0].push_back(0);

    samplein[1].push_back(0);
    samplein[1].push_back(1);
    sampleout[1].push_back(1);

    samplein[2].push_back(1);
    samplein[2].push_back(0);
    sampleout[2].push_back(1);

    samplein[3].push_back(1);
    samplein[3].push_back(1);
    sampleout[3].push_back(0);

    sample sampleInOut[4];
    for ( int i = 0; i < 4; i++)
    {
        sampleInOut[i].in = samplein[i];
        sampleInOut[i].out = sampleout[i];
    }
    vector<sample> sampleGroup(sampleInOut, sampleInOut + 4);
    testNet.training(sampleGroup, 0.0001);

    //测试数据
    vector<double> testin[4];
    vector<double> testout[4];
    testin[0].push_back(0.1);
    testin[0].push_back(0.2);
    testin[1].push_back(0.15);
    testin[1].push_back(0.9);
    testin[2].push_back(1.1);
    testin[2].push_back(0.01);
    testin[3].push_back(0.88);
    testin[3].push_back(1.03);

    sample testInOut[4];
    for ( int i = 0; i < 4; i++)
    {
        testInOut[i].in = testin[i];
    }
    vector<sample> testGroup(testInOut, testInOut + 4);

    //预测测试数据，并输出结果
    testNet.predict(testGroup);
    for ( int i = 0; i < testGroup.size(); i++)
    {
        for ( int j = 0; j < testGroup[i].in.size(); j ++)
        {
            cout << testGroup[i].in[j] << "\t";
        }
        cout << "--- prediction:";
        for ( int j = 0; j < testGroup[i].out.size(); j++)
        {
            cout << testGroup[i].out[j] << "\t";
        }
        cout << endl;
    }

    system("pause");
    return 0;
}

本文为作者原创，转载请注明出处，谢谢！

参考

• A Critical Review of Recurrent Neural Networks for Sequence Learning
• Supervised Sequence Labelling with Recurrent Neural Networks
• http://deeplearning.stanford.edu/wiki/index.php/%E5%8F%8D%E5%90%91%E4%BC%A0%E5%AF%BC%E7%AE%97%E6%B3%95
• http://blog.csdn.net/ironyoung/article/details/49455343