作为图神经网络的变压器

TL; DR：Chaitanya Joshi的文章“ Transformers are Graph Neural Networks ”的翻译：图表，公式，思想，重要链接。经作者的允许发布。

数据中心的朋友经常问同样的问题：Graph神经网络是个好主意，但是他们有没有真正的成功案例？它们有实际用途吗？

— Pinterest, Alibaba Twitter. : Transformer.

(Transformers). NLP- GNN-, , "" , .

— (representation learning).

NLP

, "" . () (latent/hidden) - . , . , (error/loss functions).

, (natural language processing; NLP), (recurrent neural networks; RNN) — , . RNN , . , RNN, .

, , RNN ( ) .

RNN NLP. : , , (attention mechanism; attention), . , — , "".

2017 , NLP — — RNN. , , !
, Yannic Kilcher.

. , $$$h$— $$$i$- $$$S$— $$$l$$$$l+1$:

$$

$$h_{i}^{\ell+1} = \text{Attention} \left( Q^{\ell} h_{i}^{\ell} \ , K^{\ell} h_{j}^{\ell} \ , V^{\ell} h_{j}^{\ell} \right),$$

$$

$$i.e.,\ h_{i}^{\ell+1} = \sum_{j \in \mathcal{S}} w_{ij} \left( V^{\ell} h_{j}^{\ell} \right),$$

$$

$$\text{} \ w_{ij} = \text{softmax}_j \left( Q^{\ell} h_{i}^{\ell} \cdot K^{\ell} h_{j}^{\ell} \right),$$

$$$j \in \mathcal{S}$, $$$Q^{\ell}, K^{\ell}, V^{\ell}$— ( Query, Key Value). , . ! — RNN, .

, :

$$$h_{i}^{\ell}$$$${ h_{j}^{\ell} ;\ \forall j \in \mathcal{S} }$$$$w_{ij}$$$$(i,j)$, softmax $$$j$. , $$$h_{i}^{\ell+1}$$$$i$, $$${ h_{j}^{\ell} }$, $$$w_{ij}$. .

(Multi-Head Attention)

- --- (dot product attention): . , "" (attention heads) () ( "" ):

$$

$$h_{i}^{\ell+1} = \text{Concat} \left( \text{head}_1, \ldots, \text{head}_K \right) O^{\ell},$$

$$

$$\text{head}_k = \text{Attention} \left( Q^{k,\ell} h_{i}^{\ell} \ , K^{k, \ell} h_{j}^{\ell} \ , V^{k, \ell} h_{j}^{\ell} \right),$$

$$$Q^{k,\ell}, K^{k,\ell}, V^{k,\ell}$— $$$k$- "" , $$$O^{\ell}$— , $$$h_i^{\ell+1}$$$$h_i^{\ell}$.

, "" " ", . .

""

Scale issues and the Feed-forward sub-layer

, - , , : . - (1), - " " , , $$$w_{ij}$. - (2), , "". $$$h_{i}^{\ell+1}$. , (normalization layer).

(2) LayerNorm, . , --- (1).

, , : . , $$$h_i^{\ell+1}$() , ReLU, , :

$$

$$h_i^{\ell+1} = \text{LN} \left( \text{MLP} \left( \text{LN} \left( h_i^{\ell+1} \right) \right) \right)$$

, , , . ! , LayerNorm . — , .
, !

:

"" , NLP- , . , ("") , (residual connections) "" "". .

GNN

NLP.

(GNN) (GCN) . (, , ). , . GNN (propagate) — — .

, , : , GNN, : "" .

GNN $$$h$$$$i$$$$\ell$$$$h_i^{\ell}$, $$$h_j^{\ell}$$$$j \in \mathcal{N}(i)$:

$$

$$h_{i}^{\ell+1} = \sigma \Big( U^{\ell} h_{i}^{\ell} + \sum_{j \in \mathcal{N}(i)} \left( V^{\ell} h_{j}^{\ell} \right) \Big),$$

$$$U^{\ell}, V^{\ell}$— GNN, $$$\sigma$— (, , ReLU). —

$$$j \in \mathcal{N}(i)$, , , / - — , .

, ?

, :

$$$j$, , Graph Attention Network (GAT). , — — "" !

— , —

, , — , . GNN, (.. ) (.. ) , .

, GNN . — $$$j \in \mathcal{N}(i)$, NLP $$$\mathcal{S}$, $$$j \in \mathcal{S}$.

, , , , , , — . GNN-. , GNN, .

?

, , , .

— , NLP?

( , ) : , . TreeLSTM, , , /GNN NLP?

(long-term dependencies)?

: . , , , $$$n$GNN $$$n^2$. - $$$n$.

NLP-, . , "" LSH (Locality-Sensitive Hashing) . .

, , GNN. , (Binary Partitioning) " ".

" "?

NLP- , , , . , — , , — - " ".

, , " ".

"" — GNN , ( , ) , GNN ? .

? ?

. , , - "" . , , .

"" GNN, GAT , MoNet (Gaussian kernels) . . "" ?

, GNN (, ) . " " !

- , ? Yann Dauphin (ConvNet). , , !

?

, , , , (learning rate schedule), "" (warmup strategy) (decay settings). , , — , — .

, , , .

DeepMind- , - ? " 16 000 "" (warmup), 500 000 "" (decay), 9 000 ".

, , , : , " " ?

" "?

, , (inductive bias), ?

, : The Illustrated Transformer The Annotated Transformer.

GNN : Arthur Szlam Attention/Memory Networks, GNN . - (position paper) DeepMind, — "" — . , , DGL. seq2seq "" "" GNN.

, , GNN NLP ( HuggingFace: Transformers).

最后，我们最近写了一篇文章，其中我们将转换器应用于具有QuickDraw sketches的数据集。一探究竟！

加成

该帖子还被翻译成中文。加入他关于reddit和Twitter的讨论！

英语翻译：Anton Alekseev
（人工智能实验室，POMI RAS以V.A. Steklov命名）

宝贵的意见，翻译感谢Denis Kiryanov柯丁 和米哈伊尔·埃夫蒂希耶夫（Mikhail Evtikhiev） aspr_spb。