libFM 的大名如雷贯耳, 然而一直没有机会看看它的具体实现; 周末查看了一下 FM 算法的原理以及源码实现, 现在记录下心得. 另外发现大佬很多, 像博主 zhiyong_will 写了一系列的文章, 比如 机器学习算法实现解析——libFM之libFM的训练过程之SGD的方法 来介绍 libFM 的源码实现, 看后受益匪浅. 感觉以后要是看其他的源码, 也应该这样来写文章, 提炼出框架中的核心功能, 将实现原理讲透彻, 代码应配合公式推导, 不需要太在意细枝末节. 另外, UML 图之类的也应该画一画.
S. Rendle. Factorization machines. In ICDM, pages995–1000, 2010.
介绍 FM 之前, 需要了解一下它被用来解决什么问题. 在推荐/搜索/CTR预估等场景, 经常会用到 Categorical 特征(即类别特征), 由于单特征的表达能力比较弱, 特征交叉是必要的, 用来增强类别特征的表达能力. 为了将类别特征转换为数值特征, 我们常用 OneHot 编码, 但由于类别特征的种类以及取值可能众多, 常导致编码后的特征空间大且稀疏. FM 模型就是用来解决稀疏数据下的特征组合问题.
作为对比, 考虑在线性模型中加入组合特征, 模型可以表示为:
y
(
x
)
=
w
0
+
∑
i
=
1
n
w
i
x
i
+
∑
i
=
1
n
∑
j
=
i
+
1
n
w
i
j
x
i
x
j
y(x) = w_0 + \sum_{i=1}^{n} w_ix_i + \sum_{i=1}^{n}\sum_{j=i+1}^{n}w_{ij}x_ix_j
y ( x ) = w 0 + i = 1 ∑ n w i x i + i = 1 ∑ n j = i + 1 ∑ n w i j x i x j
其中
n
n
n
x
x
x
x
i
x_i
x i
x
x
x
i
i
i
x
i
x
j
x_ix_j
x i x j
w
0
w_0
w 0
w
i
w_i
w i
w
i
j
w_{ij}
w i j
模型的参数的总个数为:
1
+
n
+
C
n
2
=
1
+
n
+
n
(
n
−
1
)
2
1 + n + C_n^2 = 1 + n + \frac{n(n - 1)}{2}
1 + n + C n 2 = 1 + n + 2 n ( n − 1 )
由于一般样本量大且稀疏, 二次项的权重
w
i
j
w_{ij}
w i j
x
i
x_i
x i
x
j
x_j
x j
下图来自美团技术团队的 深入FFM原理与实践
如何解决二次项参数的训练问题呢？矩阵分解提供了一种解决思路。在model-based的协同过滤中，一个rating矩阵可以分解为user矩阵和item矩阵，每个user和item都可以采用一个隐向量表示。比如在下图中的例子中，我们把每个user表示成一个二维向量，同时把每个item表示成一个二维向量，两个向量的点积就是矩阵中user对item的打分。
对于一个实对称的正定矩阵
W
W
W
W
=
V
T
V
W = V^TV
W = V T V
v
i
∈
R
k
v_i\in\mathbb{R}^{k}
v i ∈ R k
v
i
v_i
v i
V
V
V
i
i
i
v
i
v_i
v i
v
j
v_j
v j
⟨
v
i
,
v
j
⟩
\langle v_i, v_j \rangle
⟨ v i , v j ⟩
w
i
j
w_{ij}
w i j
y
(
x
)
=
w
0
+
∑
i
=
1
n
w
i
x
i
+
∑
i
=
1
n
∑
j
=
i
+
1
n
⟨
v
i
,
v
j
⟩
x
i
x
j
y(x) = w_0 + \sum_{i=1}^{n} w_ix_i + \sum_{i=1}^{n}\sum_{j=i+1}^{n}\langle v_i, v_j \rangle x_ix_j
y ( x ) = w 0 + i = 1 ∑ n w i x i + i = 1 ∑ n j = i + 1 ∑ n ⟨ v i , v j ⟩ x i x j
由于每一个特征都对应一个隐向量,
V
∈
R
k
×
n
V\in\mathbb{R}^{k\times n}
V ∈ R k × n
k
n
kn
k n
k
≪
n
k \ll n
k ≪ n
n
(
n
−
1
)
2
\frac{n(n - 1)}{2}
2 n ( n − 1 )
v
i
v_i
v i
x
i
x_i
x i
j
≠
i
j \neq i
j = i
x
i
x
j
≠
0
x_ix_j\neq 0
x i x j = 0
v
i
v_i
v i
如果直接去计算上式, 二次项的计算复杂度为
O
(
k
n
2
)
O(kn^2)
O ( k n 2 )
O
(
k
n
)
O(kn)
O ( k n )
∑
i
=
1
n
∑
j
=
i
+
1
n
⟨
v
i
,
v
j
⟩
x
i
x
j
=
1
2
∑
i
=
1
n
∑
j
=
1
n
⟨
v
i
,
v
j
⟩
x
i
x
j
−
1
2
∑
i
=
1
n
⟨
v
i
,
v
i
⟩
x
i
x
i
=
1
2
(
∑
i
=
1
n
∑
j
=
1
n
∑
f
=
1
k
v
i
,
f
v
j
,
f
x
i
x
j
−
∑
i
=
1
n
∑
f
=
1
k
v
i
,
f
v
i
,
f
x
i
x
i
)
=
1
2
∑
f
=
1
k
(
(
∑
i
=
1
n
v
i
,
f
x
i
)
(
∑
j
=
1
n
v
j
,
f
x
j
)
−
∑
j
=
1
n
v
j
,
f
2
x
j
2
)
=
1
2
∑
f
=
1
k
(
(
∑
i
=
1
n
v
i
,
f
x
i
)
2
−
∑
j
=
1
n
v
j
,
f
2
x
j
2
)
\begin{aligned} & \sum_{i=1}^{n} \sum_{j=i+1}^{n}\left\langle\mathbf{v}_{i}, \mathbf{v}_{j}\right\rangle x_{i} x_{j} \\ =& \frac{1}{2} \sum_{i=1}^{n} \sum_{j=1}^{n}\left\langle\mathbf{v}_{i}, \mathbf{v}_{j}\right\rangle x_{i} x_{j}-\frac{1}{2} \sum_{i=1}^{n}\left\langle\mathbf{v}_{i}, \mathbf{v}_{i}\right\rangle x_{i} x_{i} \\ =& \frac{1}{2}\left(\sum_{i=1}^{n} \sum_{j=1}^{n} \sum_{f=1}^{k} v_{i, f} v_{j, f} x_{i} x_{j}-\sum_{i=1}^{n} \sum_{f=1}^{k} v_{i, f} v_{i, f} x_{i} x_{i}\right) \\ =& \frac{1}{2} \sum_{f=1}^{k}\left(\left(\sum_{i=1}^{n} v_{i, f}x_{i}\right)\left(\sum_{j=1}^{n} v_{j, f} x_{j}\right) - \sum_{j=1}^{n} v^2_{j, f} x^2_{j}\right) \\ =& \frac{1}{2} \sum_{f=1}^{k}\left(\left(\sum_{i=1}^{n} v_{i, f}x_{i}\right)^2 - \sum_{j=1}^{n} v^2_{j, f} x^2_{j}\right) \\ \end{aligned}
= = = = i = 1 ∑ n j = i + 1 ∑ n ⟨ v i , v j ⟩ x i x j 2 1 i = 1 ∑ n j = 1 ∑ n ⟨ v i , v j ⟩ x i x j − 2 1 i = 1 ∑ n ⟨ v i , v i ⟩ x i x i 2 1 ⎝ ⎛ i = 1 ∑ n j = 1 ∑ n f = 1 ∑ k v i , f v j , f x i x j − i = 1 ∑ n f = 1 ∑ k v i , f v i , f x i x i ⎠ ⎞ 2 1 f = 1 ∑ k ( ( i = 1 ∑ n v i , f x i ) ( j = 1 ∑ n v j , f x j ) − j = 1 ∑ n v j , f 2 x j 2 ) 2 1 f = 1 ∑ k ⎝ ⎛ ( i = 1 ∑ n v i , f x i ) 2 − j = 1 ∑ n v j , f 2 x j 2 ⎠ ⎞
上式对
k
k
k
n
n
n
O
(
k
n
)
O(kn)
O ( k n )
(
∑
i
=
1
n
a
i
)
(
∑
j
=
1
n
b
j
)
=
∑
i
=
1
n
∑
j
=
1
n
a
i
b
j
\left(\sum_{i=1}^{n}a_i\right)\left(\sum_{j=1}^{n}b_j\right) = \sum_{i=1}^{n}\sum_{j=1}^{n}a_ib_j
( i = 1 ∑ n a i ) ( j = 1 ∑ n b j ) = i = 1 ∑ n j = 1 ∑ n a i b j
libFM 在 src/fm_core/fm_model.h 中的 predict 方法中完成对预测值
y
^
\widehat{y}
y
double fm_model:: predict ( sparse_row< FM_FLOAT> & x, DVector< double > & sum,
DVector< double > & sum_sqr) {
double result = 0 ;
if ( k0) {
result + = w0;
}
if ( k1) {
for ( uint i = 0 ; i < x. size; i++ ) {
assert ( x. data[ i] . id < num_attribute) ;
result + = w ( x. data[ i] . id) * x. data[ i] . value;
}
}
for ( int f = 0 ; f < num_factor; f++ ) {
sum ( f) = 0 ;
sum_sqr ( f) = 0 ;
for ( uint i = 0 ; i < x. size; i++ ) {
double d = v ( f, x. data[ i] . id) * x. data[ i] . value;
sum ( f) + = d;
sum_sqr ( f) + = d* d;
}
result + = 0.5 * ( sum ( f) * sum ( f) - sum_sqr ( f) ) ;
}
return result;
}
代码分为三个部分, 分别用来处理偏置
w
0
w_0
w 0
w
i
w_i
w i
v
i
v_i
v i sum(f) 来保存
∑
i
=
1
n
v
i
,
f
x
i
\sum\limits_{i=1}^{n} v_{i, f}x_{i}
i = 1 ∑ n v i , f x i sum_sqr(f) 来保存
∑
j
=
1
n
v
j
,
f
2
x
j
2
\sum\limits_{j=1}^{n} v^2_{j, f} x^2_{j}
j = 1 ∑ n v j , f 2 x j 2 注意前者在后面的参数更新中会被使用到.
以上是 FM 的基本思路, 下面介绍 FM 的目标函数以及参数(
w
0
,
w
,
V
w_0, \bm{w}, V
w 0 , w , V
FM 可以用来做回归 (Regression) 以及二分类 (Binary Classification) 任务, 其中回归的目标函数一般采用 MSE:
l
=
1
2
n
∑
i
=
1
n
(
y
i
−
y
^
i
)
2
l = \frac{1}{2n}\sum_{i=1}^{n}\left(y_i - \widehat{y}_i\right)^2
l = 2 n 1 i = 1 ∑ n ( y i − y
i ) 2
而分类任务一般用交叉熵, 但 libFM 中在实现时采用的 Log 损失函数, 参考 机器学习中的基本问题——log损失与交叉熵的等价性 , 了解到这两个损失函数是等价的, 具体的推导方法如下:
首先 Log 损失的基本形式为:
log
(
1
+
exp
(
−
y
⋅
y
^
)
)
,
y
∈
{
−
1
,
1
}
\log(1 + \exp(-y\cdot\widehat{y})), \quad y\in\{-1, 1\}
log ( 1 + exp ( − y ⋅ y
) ) , y ∈ { − 1 , 1 }
对样本集
T
=
{
(
x
1
,
y
1
)
,
…
,
(
x
n
,
y
n
)
}
\mathcal{T} = \{(x_1, y_1), \ldots, (x_n, y_n)\}
T = { ( x 1 , y 1 ) , … , ( x n , y n ) }
1
n
∑
i
=
1
n
log
(
1
+
exp
(
−
y
i
⋅
y
^
i
)
)
\frac{1}{n}\sum_{i=1}^{n}\log\left(1 + \exp(-y_i\cdot\widehat{y}_i)\right)
n 1 i = 1 ∑ n log ( 1 + exp ( − y i ⋅ y
i ) )
另一方面, Sigmoid 函数 (详见 逻辑回归模型 Logistic Regression 详细推导 (含 Numpy 与PyTorch 实现) ) 被定义为:
σ
(
x
)
=
1
1
+
exp
(
−
x
)
\sigma(x) = \frac{1}{1 + \exp(-x)}
σ ( x ) = 1 + exp ( − x ) 1
从上面公式可以推出两点结论:
⇒
1
+
exp
(
−
x
)
=
1
σ
(
x
)
⇒
σ
(
x
)
=
1
−
σ
(
−
x
)
\Rightarrow 1 + \exp(-x) = \frac{1}{\sigma(x)} \\ \Rightarrow \sigma(x) = 1 - \sigma(-x)
⇒ 1 + exp ( − x ) = σ ( x ) 1 ⇒ σ ( x ) = 1 − σ ( − x )
因此 Log 损失可以进一步推导为:
1
n
∑
i
=
1
n
log
(
1
+
exp
(
−
y
i
⋅
y
^
i
)
)
=
1
n
∑
i
=
1
n
log
(
1
σ
(
y
i
⋅
y
^
i
)
)
=
−
1
n
∑
i
=
1
n
log
(
σ
(
y
i
⋅
y
^
i
)
)
\begin{aligned} &\frac{1}{n}\sum_{i=1}^{n}\log\left(1 + \exp(-y_i\cdot\widehat{y}_i)\right) \\ &= \frac{1}{n}\sum_{i=1}^{n}\log\left(\frac{1}{\sigma\left(y_i\cdot\widehat{y}_i\right)}\right) \\ &= -\frac{1}{n}\sum_{i=1}^{n}\log\left(\sigma\left(y_i\cdot\widehat{y}_i\right)\right) \end{aligned}
n 1 i = 1 ∑ n log ( 1 + exp ( − y i ⋅ y
i ) ) = n 1 i = 1 ∑ n log ( σ ( y i ⋅ y
i ) 1 ) = − n 1 i = 1 ∑ n log ( σ ( y i ⋅ y
i ) )
上式为 libFM 中用到的损失函数 , 下面再介绍交叉熵. 交叉熵定义为:
H
(
y
,
y
^
)
=
−
1
n
∑
i
=
1
n
I
{
y
i
=
1
}
⋅
log
σ
(
y
^
i
)
+
I
{
y
i
=
−
1
}
⋅
log
(
1
−
σ
(
y
^
i
)
)
H(y, \widehat{y}) = -\frac{1}{n}\sum_{i=1}^{n}I\{y_i = 1\}\cdot\log\sigma(\widehat{y}_i) + I\{y_i = -1\}\cdot\log\left(1 - \sigma(\widehat{y}_i)\right)
H ( y , y
) = − n 1 i = 1 ∑ n I { y i = 1 } ⋅ log σ ( y
i ) + I { y i = − 1 } ⋅ log ( 1 − σ ( y
i ) )
由于
σ
(
x
)
=
1
−
σ
(
−
x
)
\sigma(x) = 1 - \sigma(-x)
σ ( x ) = 1 − σ ( − x )
H
(
y
,
y
^
)
=
−
1
n
∑
i
=
1
n
I
{
y
i
=
1
}
⋅
log
σ
(
y
^
i
)
+
I
{
y
i
=
−
1
}
⋅
log
(
σ
(
−
y
^
i
)
)
H(y, \widehat{y}) = -\frac{1}{n}\sum_{i=1}^{n}I\{y_i = 1\}\cdot\log\sigma(\widehat{y}_i) + I\{y_i = -1\}\cdot\log\left(\sigma(-\widehat{y}_i)\right)
H ( y , y
) = − n 1 i = 1 ∑ n I { y i = 1 } ⋅ log σ ( y
i ) + I { y i = − 1 } ⋅ log ( σ ( − y
i ) )
又因为:
I
{
y
(
i
)
=
k
}
=
{
0
if
y
(
i
)
≠
k
1
if
y
(
i
)
=
k
I\left\{y^{(i)}=k\right\}=\left\{ \begin{array}{ll} 0 & \text { if } y^{(i)} \neq k \\ 1 & \text { if } y^{(i)}=k \end{array}\right.
I { y ( i ) = k } = { 0 1 if y ( i ) = k if y ( i ) = k
因此, 可以得到交叉熵为:
H
(
y
,
y
^
)
=
−
1
n
∑
i
=
1
n
log
σ
(
y
i
⋅
y
^
i
)
H(y, \widehat{y}) = -\frac{1}{n}\sum_{i=1}^{n}\log\sigma(y_i\cdot\widehat{y}_i)
H ( y , y
) = − n 1 i = 1 ∑ n log σ ( y i ⋅ y
i )
该式和 Log 损失相同, 因此得到了 Log 损失和交叉熵等价的结论.
这里关注 libFM 实现的 SGD 算法, 由于每次只对一个样本
(
x
i
,
y
i
)
(x_i, y_i)
( x i , y i )
l
=
1
2
(
y
i
−
y
^
i
)
2
l = \frac{1}{2}\left(y_i - \widehat{y}_i\right)^2
l = 2 1 ( y i − y
i ) 2
其对参数的梯度为:
∂
l
∂
θ
=
−
(
y
i
−
y
^
i
)
⋅
∂
y
^
i
∂
θ
\frac{\partial l}{\partial\theta} = -\left(y_i - \widehat{y}_i\right)\cdot\frac{\partial \widehat{y}_i}{\partial\theta}
∂ θ ∂ l = − ( y i − y
i ) ⋅ ∂ θ ∂ y
i
对于回归问题, 损失函数为 Log 损失 (详见上一节):
l
=
−
log
(
σ
(
y
i
⋅
y
^
i
)
)
l = -\log(\sigma(y_i\cdot\widehat{y}_i))
l = − log ( σ ( y i ⋅ y
i ) )
梯度更新的方法为:
∂
l
∂
θ
=
−
1
σ
(
y
i
⋅
y
^
i
)
⋅
σ
′
(
y
i
⋅
y
^
i
)
⋅
y
i
⋅
∂
y
^
i
∂
θ
=
−
1
σ
(
y
i
⋅
y
^
i
)
⋅
(
σ
(
y
i
⋅
y
^
i
)
(
1
−
σ
(
y
i
⋅
y
^
i
)
)
)
⋅
y
i
⋅
∂
y
^
i
∂
θ
=
−
(
1
−
σ
(
y
i
⋅
y
^
i
)
)
⋅
y
i
⋅
∂
y
^
i
∂
θ
=
−
y
i
⋅
(
1
−
σ
(
y
i
⋅
y
^
i
)
)
⋅
∂
y
^
i
∂
θ
\begin{aligned} \frac{\partial l}{\partial\theta} &= -\frac{1}{\sigma(y_i\cdot\widehat{y}_i)}\cdot\sigma^\prime(y_i\cdot\widehat{y}_i)\cdot y_i \cdot \frac{\partial \widehat{y}_i}{\partial\theta} \\ &= -\frac{1}{\sigma(y_i\cdot\widehat{y}_i)}\cdot\left(\sigma(y_i\cdot\widehat{y}_i)\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)\right)\cdot y_i \cdot \frac{\partial \widehat{y}_i}{\partial\theta} \\ &= -\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)\cdot y_i \cdot \frac{\partial \widehat{y}_i}{\partial\theta} \\ &= -y_i\cdot \left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)\cdot \frac{\partial \widehat{y}_i}{\partial\theta} \end{aligned}
∂ θ ∂ l = − σ ( y i ⋅ y
i ) 1 ⋅ σ ′ ( y i ⋅ y
i ) ⋅ y i ⋅ ∂ θ ∂ y
i = − σ ( y i ⋅ y
i ) 1 ⋅ ( σ ( y i ⋅ y
i ) ( 1 − σ ( y i ⋅ y
i ) ) ) ⋅ y i ⋅ ∂ θ ∂ y
i = − ( 1 − σ ( y i ⋅ y
i ) ) ⋅ y i ⋅ ∂ θ ∂ y
i = − y i ⋅ ( 1 − σ ( y i ⋅ y
i ) ) ⋅ ∂ θ ∂ y
i
推导中用到了 Sigmoid 函数求导的性质:
σ
′
(
x
)
=
σ
(
x
)
(
1
−
σ
(
x
)
)
\sigma^\prime(x) = \sigma(x)(1 - \sigma(x))
σ ′ ( x ) = σ ( x ) ( 1 − σ ( x ) )
libFM 在 libfm/src/fm_learn_sgd_element.h 的 learn 方法中实现上面的更新步骤, 核心代码如下 (无关代码已删去):
void fm_learn_sgd_element:: learn ( Data& train, Data& test) {
fm_learn_sgd:: learn ( train, test) ;
for ( int i = 0 ; i < num_iter; i++ ) {
double iteration_time = getusertime ( ) ;
for ( train. data- > begin ( ) ; ! train. data- > end ( ) ; train. data- > next ( ) ) {
double p = fm- > predict ( train. data- > getRow ( ) , sum, sum_sqr) ;
double mult = 0 ;
if ( task == 0 ) {
p = std:: min ( max_target, p) ;
p = std:: max ( min_target, p) ;
mult = - ( train. target ( train. data- > getRowIndex ( ) ) - p) ;
} else if ( task == 1 ) {
mult = - train. target ( train. data- > getRowIndex ( ) ) * ( 1.0 - 1.0 / ( 1.0 + exp ( - train. target ( train. data- > getRowIndex ( ) ) * p) ) ) ;
}
SGD ( train. data- > getRow ( ) , mult, sum) ;
}
}
}
其中
double p = fm- > predict ( train. data- > getRow ( ) , sum, sum_sqr) ;
完成预估值的计算 (前面在阐述 “基本思路” 时已经介绍过了):
y
^
(
x
)
=
w
0
+
∑
i
=
1
n
w
i
x
i
+
∑
i
=
1
n
∑
j
=
i
+
1
n
⟨
v
i
,
v
j
⟩
x
i
x
j
=
w
0
+
∑
i
=
1
n
w
i
x
i
+
1
2
∑
f
=
1
k
(
(
∑
i
=
1
n
v
i
,
f
x
i
)
2
−
∑
j
=
1
n
v
j
,
f
2
x
j
2
)
\begin{aligned} \widehat{y}(x) &= w_0 + \sum_{i=1}^{n} w_ix_i + \sum_{i=1}^{n}\sum_{j=i+1}^{n}\langle v_i, v_j \rangle x_ix_j \\ &= w_0 + \sum_{i=1}^{n} w_ix_i + \frac{1}{2} \sum_{f=1}^{k}\left(\left(\sum_{i=1}^{n} v_{i, f}x_{i}\right)^2 - \sum_{j=1}^{n} v^2_{j, f} x^2_{j}\right) \end{aligned}
y
( x ) = w 0 + i = 1 ∑ n w i x i + i = 1 ∑ n j = i + 1 ∑ n ⟨ v i , v j ⟩ x i x j = w 0 + i = 1 ∑ n w i x i + 2 1 f = 1 ∑ k ⎝ ⎛ ( i = 1 ∑ n v i , f x i ) 2 − j = 1 ∑ n v j , f 2 x j 2 ⎠ ⎞
p 即为
y
^
i
\widehat{y}_i
y
i sum(f) 来保存
∑
i
=
1
n
v
i
,
f
x
i
\sum\limits_{i=1}^{n} v_{i, f}x_{i}
i = 1 ∑ n v i , f x i mult 表示
−
(
y
i
−
y
^
i
)
-\left(y_i - \widehat{y}_i\right)
− ( y i − y
i ) mult 表示
−
y
i
⋅
(
1
−
σ
(
y
i
⋅
y
^
i
)
)
-y_i\cdot\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)
− y i ⋅ ( 1 − σ ( y i ⋅ y
i ) )
之后的 SGD 函数, 真正实现对参数
(
w
0
,
w
,
V
)
(w_0, \bm{w}, V)
( w 0 , w , V )
∂
y
^
i
∂
θ
\frac{\partial \widehat{y}_i}{\partial\theta}
∂ θ ∂ y
i
θ
\theta
θ
∂
∂
θ
y
^
i
(
x
)
=
{
1
,
if
θ
is
w
0
x
i
,
if
θ
is
w
i
x
i
∑
j
=
1
n
v
j
,
f
x
j
−
v
i
,
f
x
i
2
,
if
θ
is
v
i
,
f
\frac{\partial}{\partial \theta} \hat{y}_i(\mathbf{x})=\left\{ \begin{array}{ll} 1, & \text { if } \theta \text { is } w_{0} \\ x_{i}, & \text { if } \theta \text { is } w_{i} \\ x_{i} \sum\limits_{j=1}^{n} v_{j, f} x_{j}-v_{i, f} x_{i}^{2}, & \text { if } \theta \text { is } v_{i, f} \end{array}\right.
∂ θ ∂ y ^ i ( x ) = ⎩ ⎪ ⎪ ⎨ ⎪ ⎪ ⎧ 1 , x i , x i j = 1 ∑ n v j , f x j − v i , f x i 2 , if θ is w 0 if θ is w i if θ is v i , f
需要注意的是, 在前面计算
y
^
i
(
x
)
\widehat{y}_i(x)
y
i ( x ) sum(f) 保存了
∑
j
=
1
n
v
j
,
f
x
j
\sum\limits_{j=1}^{n} v_{j, f} x_{j}
j = 1 ∑ n v j , f x j
θ
\theta
θ
O
(
1
)
O(1)
O ( 1 )
SGD 函数定义在 src/fm_core/fm_sgd.h 中, 具体为:
void fm_SGD ( fm_model* fm, const double & learn_rate, sparse_row< DATA_FLOAT> & x,
const double multiplier, DVector< double > & sum) {
if ( fm- > k0) {
double & w0 = fm- > w0;
w0 - = learn_rate * ( multiplier + fm- > reg0 * w0) ;
}
if ( fm- > k1) {
for ( uint i = 0 ; i < x. size; i++ ) {
double & w = fm- > w ( x. data[ i] . id) ;
w - = learn_rate * ( multiplier * x. data[ i] . value + fm- > regw * w) ;
}
}
for ( int f = 0 ; f < fm- > num_factor; f++ ) {
for ( uint i = 0 ; i < x. size; i++ ) {
double & v = fm- > v ( f, x. data[ i] . id) ;
double grad = sum ( f) * x. data[ i] . value - v * x. data[ i] . value * x. data[ i] . value;
v - = learn_rate * ( multiplier * grad + fm- > regv * v) ;
}
}
}
代码中还考虑了对各个参数的正则项. 其中 multiplier 就是前面 mult, 对于回归问题, 表示
−
(
y
i
−
y
^
i
)
-\left(y_i - \widehat{y}_i\right)
− ( y i − y
i ) multiplier 表示
−
y
i
⋅
(
1
−
σ
(
y
i
⋅
y
^
i
)
)
-y_i\cdot\left(1 - \sigma(y_i\cdot\widehat{y}_i)\right)
− y i ⋅ ( 1 − σ ( y i ⋅ y
i ) ) sum(f) 表示
∑
j
=
1
n
v
j
,
f
x
j
\sum\limits_{j=1}^{n} v_{j, f} x_{j}
j = 1 ∑ n v j , f x j
关于 libFM 就分析到这了~
