Self-attention and the Transformer - cutting the sequential chain and referencing directly
Where an RNN passed context along by accumulating it through previous states, the Transformer looks up the tokens it needs directly. How self-attention removes the sequential structure, and how it connects back to attention Seq2Seq.
Recap
Part three boiled the remaining problems of attention Seq2Seq down to two.
- Long-range dependencies on the output side: the decoder’s $S_k$ still depends on $S_{k-1}$
- No parallelisation: both $h_k$ and $S_k$ can only be computed once the previous state exists
Both share a single cause. An RNN-based architecture passes context along by accumulating it in order.
From attention Seq2Seq to self-attention
Part two flagged one thing: in attention Seq2Seq, $K$ and $V$ are both the same variable, $h_i$.
| Concept | Attention Seq2Seq |
|---|---|
| $Q$ | decoder state $S_k$ |
| $K$ | encoder state $h_i$ |
| $V$ | encoder state $h_i$ |
$K = V = h_i$ already held. Self-attention takes one more step: $Q$ comes from the same sequence too.
| Concept | Self-attention |
|---|---|
| $Q$ | from token $x_i$ |
| $K$ | from token $x_j$ |
| $V$ | from token $x_j$ |
It is a shift from “attention that references another sequence” to “attention that references itself”.
How self-attention works
Each token $x_i$ is transformed into $q_i, k_i, v_i$ through learned weight matrices.
\[q_i = W_Q x_i, \quad k_i = W_K x_i, \quad v_i = W_V x_i\]From there the computation is identical to the attention in part two. Token $i$ computes relevance against every other token $j$,
\[w_{ij} = \text{softmax}\!\left(\frac{\langle q_i, k_j \rangle}{\sqrt{d_k}}\right)\]and sums $v_j$ with those weights to build a new representation.
\[h_i^{\text{new}} = \sum_j w_{ij} \cdot v_j\]Scaling by $\sqrt{d_k}$ prevents the inner products from growing with dimension and making the softmax excessively sharp.
The sequential structure disappears
In an RNN, for $h_4$ to get information from $h_1$ it had to pass through $h_2$ and $h_3$ in order.
In self-attention, computing $h_4^{\text{new}}$ references $x_1$ directly. No intermediate states are needed. The relevance of every token pair is computed in one go.
This structure solves both problems from part three at once.
- Long-range dependencies: any token references any other token directly, in one step
- Parallelisation: computing each token’s $q_i, k_i, v_i$ and the attention operation has no order dependency
The structure of the Transformer
The Transformer applies self-attention on both the encoder and decoder side.
Encoder: performs self-attention among the input tokens. Each token updates its own representation by directly referencing every token in the input sequence.
Decoder: uses two kinds of attention.
- Masked self-attention: performs self-attention among the previous output tokens, but masks out future tokens so they cannot be referenced. At generation step $k$, only $y_1 \sim y_{k-1}$ are available.
- Cross-attention: structurally identical to the attention in part two. $Q$ comes from the decoder state, $K$ and $V$ from the encoder output.
Positional encoding
An RNN reads tokens in order, so positional information is inherent in the structure. Self-attention sees every token at once, so there is no positional information. It cannot tell “I love cats” from “cats love I”.
To fix this, positional information is added to each token’s embedding.
\[\tilde{x}_i = x_i + \text{PE}(i)\]The original paper proposed a fixed positional encoding using sin/cos functions; learned variants are also widely used now.
Conclusion to the series
The path from Seq2Seq to the Transformer summarises like this.
| Architecture | Key characteristic | Remaining problem |
|---|---|---|
| Vanilla Seq2Seq | depends on the encoder’s final state | encoder information bottleneck |
| Attention Seq2Seq | directly references all encoder states | decoder sequentiality, no parallelisation |
| Transformer | direct reference in both encoder and decoder | none |
The change at each step was not simply a move to a more complex model. It was a structural shift in how information is passed.
Where an RNN passed context along by accumulating it in order, the Transformer looks up the context it needs directly.