以Word2Vec为例的错误反向传播算法

由于在寻找所需的错误的反向传播机制的解释时遇到了很大的困难，因此，我决定使用Word2Vec算法撰写有关错误的反向传播的文章。我的目标是使用简单但非平凡的神经网络来解释算法的本质。另外，word2vec在NLP社区中变得如此流行，以至于关注它会很有用。

这篇文章与我建议阅读的另一篇更实用的文章有关，它讨论了在Python中直接实现word2vec的方法。在这篇文章中，我们将主要集中在理论部分。

让我们从真正了解反向传播所需的东西开始。除了来自机器学习的概念（例如损失函数和梯度下降）之外，数学中的另外两个组件也派上用场：

• 线性代数（特别是矩阵乘法）
• 功能与许多变量的微分链的规则

如果您熟悉这些概念，那么进一步的考虑将很简单。如果尚未掌握它们，您仍然可以了解反向传播的基础知识。

首先，我要定义反向传播的概念，如果含义不够清楚，则将在以下段落中更详细地公开它。

1.什么是反向传播算法？

在神经网络的框架内，训练网络（即最小化损失函数）所涉及的唯一参数是权重（这里我指的是广义上的权重，指代权重和偏差）。权重在每次迭代中都会变化，直到我们达到损失函数的最小值为止。

, — , , .
, , .
, , , , w1 w2.

1. .

, w1 w2 .

, . , $$$\partial\mathcal{L}/\partial w_1$ $$$\partial\mathcal{L}/\partial w_2$, , . $$$\eta$, .

2. Word2Vec

word2vec, , , . , word2vec, NLP.

, word2vec [N, 3], N - , . , , '', , ( ), , ''. , word2vec .

word2vec : (CBOW) (skip-gram). , CBOW, , skip-gram.

. , woed2vec .

3. CBOW

CBOW . , :

2. Continuous Bag-of-Words

, ,
a = 1 (identity function, , ).
Softmax.

one hot encoding , , , , , 1.

: ['', '', '', '', '', '']
OneHot('') = [0, 0, 0, 1, 0, 0]
OneHot(['', '']) = [1, 0, 0, 1, 0, 0]
OneHot(['', '', '']) = [1, 0, 0, 0, 1, 1]

, W $$$V\times N$, $$$W’$ $$$N\times V$, V — , N — ( , word2vec)

y t, , , , , .

, .

3.1 (Loss function)

1, , x:

$$

$$\begin{eqnarray*} \textbf{h} = & W^T\textbf{x} \hspace{7.0cm} \\ \textbf{u}= & W'^T\textbf{h}=W'^TW^T\textbf{x} \hspace{4.6cm} \\ \textbf{y}= & \ \ \mathbb{S}\textrm{oftmax}(\textbf{u})= \mathbb{S}\textrm{oftmax}(W'^TW^T\textbf{x}) \hspace{2cm} \end{eqnarray*}$$

, h — , u — , y — .

, , , (wt, wc). , onehot encoding .

, onehot wt ( ).

softmax , :

$$

$$\begin{equation*} \mathcal{L} = -\log \mathbb{P}(w_t|w_c)=-\log y_{j^*}\ =-\log[\mathbb{S}\textrm{oftmax}(u_{j^*})]=-\log\left(\frac{\exp{u_{j^*}}}{\sum_i \exp{u_i}}\right), \end{equation*}$$

, j* — .
. (1):

$$

$$\begin{equation} \bbox[lightblue,5px,border:2px solid red]{ \mathcal{L} = -u_{j^*} + \log \sum_i \exp{(u_i)}. } \label{eq:loss} \end{equation} \>(1)$$

.

3.2 CBOW

, , W W` . , .

. (1) W W`. $$$\partial \mathcal{L}/\partial{W}$ $$$\partial \mathcal{L}/\partial{W’}$

, . (1) W W`, u=[u1, ...., uV],

$$

$$\begin{equation*} \mathcal{L} = \mathcal{L}(\mathbf{u}(W,W'))=\mathcal{L}(u_1(W,W'), u_2(W,W'),\dots, u_V(W,W'))\ . \end{equation*}$$

:

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \sum_{k=1}^V\frac{\partial\mathcal{L}}{\partial u_k}\frac{\partial u_k}{\partial W'_{ij}} \label{eq:dLdWp} \end{equation} \>(2)$$

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V\frac{\partial\mathcal{L}}{\partial u_k}\frac{\partial u_k}{\partial W_{ij}}\ . \label{eq:dLdW} \end{equation} \>(3)$$

, (2) (3) .

(2), Wij, W, i j , uj ( yj).

3. (a) $$$y_j$ $$$h_i$ $$$W'_{ij}$ $$$W'$. (b) , $$$x_k$ N $$$W_{k1}\dots W_{kN}$ W.

, $$$\partial u_k/\partial W'_{ij}$, , k=j, 0.

(4):

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \frac{\partial\mathcal{L}}{\partial u_j}\frac{\partial u_j} {\partial W'_{ij}} \label{eq:derivative#1} \end{equation} \>(4)$$

$$$\partial \mathcal{L}/\partial u_j$, (5):

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial u_j} = -\delta_{jj^*} + y_j := e_j \label{eq:term#1} \end{equation} \>(5)$$

, $$$\delta_{jj^*}$ — , , 1, , 0 .

(5) e N ( ), , , .

(4) (6):

$$

$$\begin{equation} \frac{\partial u_j}{\partial W'_{ij}} = \sum_{k=1}^V W_{ik}x_k \label{eq:term#2} \end{equation} \>(6)$$

(5) (6) (4) (7):

$$

$$\begin{equation} \bbox[white,5px,border:2px dotted red]{ \frac{\partial\mathcal{L}}{\partial W'_{ij}} = (-\delta_{jj^*} + y_j) \left(\sum_{k=1}^V W_{ki}x_k\right) } \label{eq:backprop1} \end{equation} \>(7)$$

$$$\partial\mathcal{L}/\partial W_{ij}$, Xk, yj j W , 3(b). . $$$\partial u_k/\partial W_{ij}$, uk u :

$$

$$\begin{equation*} u_k = \sum_{m=1}^N\sum_{l=1}^VW'_{mk}W_{lm}x_l\ . \end{equation*}$$

$$$\partial u_k/\partial W_{ij}$, l=i m=j, (8):

$$

$$\begin{equation} \frac{\partial u_k}{\partial W_{ij}} = W'_{jk}x_i\ . \label{eq:term#3} \end{equation} \>(8)$$

(5) (8) , (9):

$$

$$\begin{equation} \bbox[white,5px,border:2px dotted red]{ \frac{\partial \mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V (-\delta_{kk^*}+y_k)W'_{jk}x_i } \label{eq:backprop2} \end{equation} \>(9)$$

3.3

, (7) (9), , . . $$$\eta>0$, :

$$

$$\begin{eqnarray} W_{\textrm{new}} & = W_{\textrm{old}} - \eta \frac{\partial \mathcal{L}}{\partial W} \nonumber \\ W'_{\textrm{new}} & = W'_{\textrm{old}} - \eta \frac{\partial \mathcal{L}}{\partial W'} \nonumber \\ \end{eqnarray}$$

3.4

. , . , . , . , , , .

4. CBOW

CBOW . . (4) . OneHot Encoded . word2vec. .

4. CBOW

CBOW CBOW .

$$

$$\begin{eqnarray} \textbf{h} = & \frac{1}{C} W^T \sum_{c=1}^C\textbf{x}^{(c)} = W^T\overline{\textbf{x}}\hspace{5.8cm} \nonumber \\ \textbf{u}= & W'^T\textbf{h}= \frac{1}{C}\sum_{c=1}^CW'^T W^T\textbf{x}^{(c)}=W'^T W^T\overline{\textbf{x}} \hspace{2.8cm} \nonumber \\ \textbf{y}= & \ \ \mathbb{S}\textrm{oftmax}(\textbf{u})= \mathbb{S}\textrm{oftmax}\left( W'^T W^T\overline{\textbf{x}}\right) \hspace{3.6cm} \nonumber \end{eqnarray}$$

, '' $$$\overline{\textbf{x}}=\sum_{c=1}^C\textbf{x}^{(c)}/C$

, . :

$$

$$\begin{equation} \mathcal{L} = -\log\mathbb{P}(w_o|w_{c,1},w_{c,2},\dots,w_{c,C})=-u_{j^*} + \log \sum_i \exp{(u_i)}. \end{equation} \>(12)$$

, :

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \sum_{k=1}^V\frac{\partial\mathcal{L}}{\partial u_k}\frac{\partial u_k}{\partial W'_{ij}} \end{equation} \>(13)$$

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V\frac{\partial\mathcal{L}}{\partial u_k}\frac{\partial u_k}{\partial W_{ij}}\ . \end{equation} \>(14)$$

CBOW , , . $$$W’_{ij}$

$$∂L∂W′ij=∑Vk=1∂L∂uk∂uk∂W′ij=∂L∂uj∂uj∂W′ij=(−δjj∗+yj)(∑Vk=1Wki¯xk)(15)$\begin{equation} \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \sum_{k=1}^V\frac{\partial\mathcal{L}}{\partial u_k}\frac{\partial u_k}{\partial W'_{ij}} = \frac{\partial\mathcal{L}}{\partial u_j}\frac{\partial u_j}{\partial W'_{ij}} = (-\delta_{jj^*} + y_j) \left(\sum_{k=1}^V W_{ki}\overline{x}_k\right) \end{equation} \> (15)$

$$$W_{ij}$:

$$

$$\begin{equation} \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V\frac{\partial\mathcal{L}}{\partial u_k}\frac{\partial}{\partial W_{ij}}\left(\frac{1}{C}\sum_{m=1}^N\sum_{l=1}^V W'_{mk}\sum_{c=1}^C W_{lm}x_l^{(c)}\right)=\frac{1}{C}\sum_{k=1}^V\sum_{c=1}^C(-\delta_{kk^*} + y_k)W'_{jk}x_i^{(c)} . \end{equation} \>(16)$$

:

$$

$$\begin{equation} \bbox[white,5px,border:2px dotted red]{ \frac{\partial\mathcal{L}}{\partial W'_{ij}} = (-\delta_{jj^*} + y_j) \left(\sum_{k=1}^V W_{ki}\overline{x}_k\right) } \label{eq:backprop1_multi} \end{equation} \>(17)$$

$$

$$\begin{equation} \bbox[white,5px,border:2px dotted red]{ \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V(-\delta_{kk^*} + y_k)W'_{jk}\overline{x}_i . } \label{eq:backprop2_multi} \end{equation} \> (18)$$

5. Skip-gram

CBOW, , . :

5. Skip-gram .

skip-gram :

$$

$$\begin{eqnarray*} \textbf{h} = & W^T\textbf{x} \hspace{9.4cm} \\ \textbf{u}_c= & W'^T\textbf{h}=W'^TW^T\textbf{x} \hspace{4cm} c=1, \dots, C \hspace{0.7cm}\\ \textbf{y}_c = & \ \ \mathbb{S}\textrm{oftmax}(\textbf{u})= \mathbb{S}\textrm{oftmax}(W'^TW^T\textbf{x}) \hspace{2cm} c=1, \dots, C \end{eqnarray*}$$

( $$$\textbf{u}_c$) , $$$\mathbf{y}_1=\mathbf{y}_2\dots= \mathbf{y}_C$. :

$$

$$\begin{eqnarray*} \mathcal{L} = -\log \mathbb{P}(w_{c,1}, w_{c,2}, \dots, w_{c,C}|w_o)=-\log \prod_{c=1}^C \mathbb{P}(w_{c,i}|w_o) \\ = -\log \prod_{c=1}^C \frac{\exp(u_{c,j^*})}{\sum_{j=1}^V \exp(u_{c,j})} =-\sum_{c=1}^C u_{c,j^*} + \sum_{c=1}^C \log \sum_{j=1}^V \exp(u_{c,j}) \end{eqnarray*}$$

skip-gram $$$C\times V$
:

$$

$$\begin{equation*} \mathcal{L} = \mathcal{L}(\mathbf{u_1}(W,W'), \mathbf{u_2}(W,W'), \dots, \mathbf{u_C}(W,W'))\\=\mathcal{L}(u_{1,1}(W,W'), u_{1,2}(W,W'), \dots, u_{C,V}(W,W')) \end{equation*}$$

:

$$

$$\begin{equation*} \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \sum_{k=1}^V\sum_{c=1}^C\frac{\partial\mathcal{L}}{\partial u_{c,k}}\frac{\partial u_{c,k}}{\partial W'_{ij}} \end{equation*}$$

$$

$$\begin{equation*} \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V\sum_{c=1}^C\frac{\partial\mathcal{L}}{\partial u_{c,k}}\frac{\partial u_{c,k}}{\partial W_{ij}}\ . \end{equation*}$$

$$$\partial \mathcal{L}/\partial u_{c,j}$, :

$$

$$\begin{equation*} \frac{\partial\mathcal{L}}{\partial u_{c,j}} = -\delta_{jj_c^*} + y_{c,j} := e_{c,j} \end{equation*}$$

CBOW :

$$

$$\begin{equation*} \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \sum_{k=1}^V\sum_{c=1}^C\frac{\partial\mathcal{L}}{\partial u_{c,k}}\frac{\partial u_{c,k}}{\partial W'_{ij}} = \sum_{c=1}^C\frac{\partial\mathcal{L}}{\partial u_{c,j}}\frac{\partial u_{c,j}}{\partial W'_{ij}} = \sum_{c=1}^C(-\delta_{jj_c^*} + y_{c,j}) \left(\sum_{k=1}^V W_{ki}x_k\right) \end{equation*}$$

$$$W_{ij}$ , :

$$

$$\begin{equation*} \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V\sum_{c=1}^C\frac{\partial\mathcal{L}}{\partial u_{c,k}}\frac{\partial}{\partial W_{ij}}\left(\sum_{m=1}^N\sum_{l=1}^V W'_{mk} W_{lm}x_l\right)=\sum_{k=1}^V\sum_{c=1}^C (-\delta_{kk_c^*} + y_{c,k})W'_{jk}x_i . \end{equation*}$$

, skip-gram :

$$

$$\begin{equation} \bbox[white,5px,border:2px dotted red]{ \frac{\partial\mathcal{L}}{\partial W'_{ij}} = \sum_{c=1}^C(-\delta_{jj_c^*} + y_{c,j}) \left(\sum_{k=1}^V W_{ki}x_k\right) } \label{eq:backprop1_skip} \end{equation} \>(21)$$

$$

$$\begin{equation} \bbox[white,5px,border:2px dotted red]{ \frac{\partial\mathcal{L}}{\partial W_{ij}} = \sum_{k=1}^V\sum_{c=1}^C (-\delta_{kk_c^*} + y_{c,k})W'_{jk}x_i . } \label{eq:backprop2_skip} \end{equation} \>(22)$$

6.

word2vec. . [2] ( softmax, negative sampling), . [1].

, word2vec.

下一步是用您喜欢的编程语言实现这些方程式。如果您喜欢Python，我已经在下一篇文章中实现了这些等式。

希望在那里见到你！

网站连结

[1] X. Rong，《word2vec参数学习解释》，arXiv：1411.2738（2014）。
[2] T. Mikolov，K。Chen，G。Corrado，J。Dean，向量空间中单词表示的有效估计，arXiv：1301.3781（2013年）。