The decoder's remaining problem - output context still accumulates
Attention solved the encoder-side information bottleneck. But the decoder itself is still a sequential structure that depends on its previous state. Here is why that becomes the final motivation for the Transformer.
Recap
In part two, attention solved the encoder-side problem. The decoder sees all of $h_1$ through $h_n$ and computes $c_k$ dynamically according to its current state $S_k$. The bottleneck at a single final state is gone.
But part two left a problem hanging.
\[S_k = f(S_{k-1},\; \text{emb}(y_{k-1}),\; c_k)\]$c_k$ is computed dynamically by attention, but producing $S_k$ still requires $S_{k-1}$.
The decoder is still sequential
In attention Seq2Seq, the decoder’s state transitions flow like this.
\[S_1 \rightarrow S_2 \rightarrow S_3 \rightarrow \cdots \rightarrow S_k\]Computing $S_k$ requires $S_{k-1}$, and computing $S_{k-1}$ requires $S_{k-2}$. The context of the output sequence is passed along, accumulated in order, through the previous decoder states.
This is essentially the same structure the encoder had in vanilla Seq2Seq.
Problem 1: long-range dependencies on the output side
Suppose we are generating “I walked a long way with a friend yesterday and finally arrived.” At the moment we generate “arrived”, the information about “I” was in $S_1$. For that information to reach $S_k$, it has to pass through every intermediate state.
\[S_1 \rightarrow S_2 \rightarrow \cdots \rightarrow S_k\]Just as in the encoder, the longer the output, the more the information of early output tokens can be diluted in the current state. An LSTM’s forget gate mitigates this to a degree, but the limit of a fundamentally sequential structure remains.
Problem 2: no parallelisation
The more practical problem is parallelisation.
$S_k$ depends on $S_{k-1}$, and $S_{k-1}$ depends on $S_{k-2}$. In other words, each state of the output sequence must be computed strictly in order. You cannot compute $S_3$ and $S_7$ at the same time.
The same goes for the encoder. Computing $h_k$ requires $h_{k-1}$ first. An RNN-based architecture cannot avoid a chain of sequential computation as long as the sequence.
This is not merely a question of speed. Training a model on long sequences over large data requires parallel computation. Without it you never properly exploit a GPU’s parallelism.
What attention fixed and what it left behind
To summarise, the attention Seq2Seq architecture looks like this.
| Vanilla Seq2Seq | Attention Seq2Seq | |
|---|---|---|
| Access to encoder information | a single $h_T$ | direct reference to $h_1 \sim h_n$ |
| Encoder sequentiality | yes | yes |
| Decoder sequentiality | yes | yes |
| Parallelisation | no | no |
Attention changed how information is accessed, but it left the sequential structure of both encoder and decoder untouched.
The core point is this.
Attention Seq2Seq reduced the dependence on the encoder’s final state, but the decoder still depends on its previous state.
Coming up next
So what if we remove the sequential structure itself? What if we drop the RNN and let every token reference every other token directly?
That is the core idea of the Transformer. Attention is applied not only between encoder and decoder, but within the sequence itself. This is called self-attention.