transformer

From Seq2Seq to Transformer: from accumulated context to direct reference · Part 4

Self-attention and the Transformer - cutting the sequential chain and referencing directly

Where an RNN passed context along by accumulating it through previous states, the Transformer looks up the tokens it needs directly. How self-attention removes the sequential structure, and how it connects back to attention Seq2Seq.

transformer  self-attention  multi-head-attention  positional-encoding  masked-attention

deep-learning

2023-07-14   ·   4 min read