转自 http://m.blog.csdn.net/u013010889/article/details/76343758
一直对softmax的反向传播的caffe代码看不懂,最近在朱神的数学理论支撑下给我详解了它的数学公式,才豁然开朗
Softmax公式推导:
caffe源码
// top_diff是下一层传过来的梯度,bottom_diff是该层往前反传的梯度
// top_data是该层输出到下一层的结果
template <typename Dtype>
void SoftmaxLayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
const vector<bool>& propagate_down,
const vector<Blob<Dtype>*>& bottom) {
const Dtype* top_diff = top[0]->cpu_diff();
const Dtype* top_data = top[0]->cpu_data();
Dtype* bottom_diff = bottom[0]->mutable_cpu_diff();
Dtype* scale_data = scale_.mutable_cpu_data();
int channels = top[0]->shape(softmax_axis_);
int dim = top[0]->count() / outer_num_;
// bottom_diff = top_diff而top_diff是dloss/da(见我手写的公式推导) shape: Cx1
caffe_copy(top[0]->count(), top_diff, bottom_diff);
for (int i = 0; i < outer_num_; ++i) {
// compute dot(top_diff, top_data) and subtract them from the bottom diff
// dloss/da和a的内积(见我手写的公式推导),scale_data保存了该内积
for (int k = 0; k < inner_num_; ++k) {
scale_data[k] = caffe_cpu_strided_dot<Dtype>(channels,
bottom_diff + i * dim + k, inner_num_,
top_data + i * dim + k, inner_num_);
}
// subtraction
// sum_multiplier_.cpu_data()由Reshape函数定义了该向量,shape: C×1,值都为1
// 作用是把dloss/da和a的内积这个标量变成Cx1的行向量
// bottom_diff = -1*sum_multiplier_.cpu_data()*scale_data+bottom_diff 大括号里的减法
caffe_cpu_gemm<Dtype>(CblasNoTrans, CblasNoTrans, channels, inner_num_, 1,
-1., sum_multiplier_.cpu_data(), scale_data, 1., bottom_diff + i * dim);
}
// elementwise multiplication
// 大括号外的对应元素相乘
caffe_mul(top[0]->count(), bottom_diff, top_data, bottom_diff);
}
SoftmaxWithLoss公式推导:
caffe源码
template <typename Dtype>
void SoftmaxWithLossLayer<Dtype>::Backward_cpu(const vector<Blob<Dtype>*>& top,
const vector<bool>& propagate_down, const vector<Blob<Dtype>*>& bottom) {
if (propagate_down[1]) {
LOG(FATAL) << this->type()
<< " Layer cannot backpropagate to label inputs.";
}
if (propagate_down[0]) {
Dtype* bottom_diff = bottom[0]->mutable_cpu_diff();
const Dtype* prob_data = prob_.cpu_data();
// 梯度全部设为: ak(见我手写的公式推导)
caffe_copy(prob_.count(), prob_data, bottom_diff);
const Dtype* label = bottom[1]->cpu_data();
int dim = prob_.count() / outer_num_;
int count = 0;
for (int i = 0; i < outer_num_; ++i) {
for (int j = 0; j < inner_num_; ++j) {
const int label_value = static_cast<int>(label[i * inner_num_ + j]);
// 设置ignor_label的地方,梯度设为0
if (has_ignore_label_ && label_value == ignore_label_) {
for (int c = 0; c < bottom[0]->shape(softmax_axis_); ++c) {
bottom_diff[i * dim + c * inner_num_ + j] = 0;
}
} else {
// 在k==y的地方把梯度改为: ak-1(见我手写的公式推导)
bottom_diff[i * dim + label_value * inner_num_ + j] -= 1;
++count;
}
}
}
// Scale gradient
Dtype loss_weight = top[0]->cpu_diff()[0] /
get_normalizer(normalization_, count);
caffe_scal(prob_.count(), loss_weight, bottom_diff);
}
}
Softmax注意点
Softmax前传时有求指数的操作,如果z很小或者很大,很容易发生float/double的上溢和下溢。这个问题其实也是有解决办法的,caffe源码中求 exponential 之前将z的每一个元素减去z分量中的最大值。这样求 exponential 的时候会碰到的最大的数就是 0 了,不会发生 overflow 的问题,但是如果其他数原本是正常范围,现在全部被减去了一个非常大的数,于是都变成了绝对值非常大的负数,所以全部都会发生 underflow,但是 underflow 的时候得到的是 0,这其实是非常 meaningful 的近似值,而且后续的计算也不会出现奇怪的 NaN。
详情参考这篇博客Softmax vs. Softmax-Loss: Numerical Stability
template <typename Dtype>
void SoftmaxLayer<Dtype>::Forward_cpu(const vector<Blob<Dtype>*>& bottom,
const vector<Blob<Dtype>*>& top) {
const Dtype* bottom_data = bottom[0]->cpu_data();
Dtype* top_data = top[0]->mutable_cpu_data();
Dtype* scale_data = scale_.mutable_cpu_data();
int channels = bottom[0]->shape(softmax_axis_);
int dim = bottom[0]->count() / outer_num_;
caffe_copy(bottom[0]->count(), bottom_data, top_data);
// We need to subtract the max to avoid numerical issues, compute the exp,
// and then normalize.
for (int i = 0; i < outer_num_; ++i) {
// initialize scale_data to the first plane
// 计算z分量中的最大值
caffe_copy(inner_num_, bottom_data + i * dim, scale_data);
for (int j = 0; j < channels; j++) {
for (int k = 0; k < inner_num_; k++) {
scale_data[k] = std::max(scale_data[k],
bottom_data[i * dim + j * inner_num_ + k]);
}
}
// subtraction
caffe_cpu_gemm<Dtype>(CblasNoTrans, CblasNoTrans, channels, inner_num_,
1, -1., sum_multiplier_.cpu_data(), scale_data, 1., top_data);
// exponentiation
caffe_exp<Dtype>(dim, top_data, top_data);
// sum after exp
caffe_cpu_gemv<Dtype>(CblasTrans, channels, inner_num_, 1.,
top_data, sum_multiplier_.cpu_data(), 0., scale_data);
// division
for (int j = 0; j < channels; j++) {
caffe_div(inner_num_, top_data, scale_data, top_data);
top_data += inner_num_;
}
}
}
参考博客
深度学习笔记8:softmax层的实现
Caffe Softmax层的实现原理?