Vanilla Seq2Seq - the architecture that leaned on one final state
To understand the basic Seq2Seq architecture you first have to accept one fact: the input is compressed into a single vector. How everything ends up in the last hidden state, and the fundamental limits that follow.
Introduction
This series traces the evolution from Seq2Seq to the Transformer, looking at what problem existed at each step and how it was solved. There is a single theme: how do you pass information along?
Part one covers the most basic form, vanilla Seq2Seq.
Encoder and decoder
A Seq2Seq model splits into two parts.
- Encoder: reads the input sentence and compresses it into a single vector
- Decoder: takes that vector and generates the output sentence one word at a time
Both parts are usually built from RNN, LSTM, or GRU cells.
Accumulating context: how the encoder works
The encoder reads the input tokens in order, updating a hidden state. Looking a little closer, the encoder cell (RNN/LSTM/GRU) takes two inputs at every step: the previous hidden state $h_{t-1}$ (the context read so far) and the current token embedding $\text{emb}(x_t)$ (the word being read now).
\[h_t = \text{encoder}(h_{t-1},\; \text{emb}(x_t))\]Unrolled in order for inputs $x_1, x_2, x_3$:
\[h_1 = \text{encoder}(h_0,\; \text{emb}(x_1))\] \[h_2 = \text{encoder}(h_1,\; \text{emb}(x_2))\] \[h_3 = \text{encoder}(h_2,\; \text{emb}(x_3))\]The first step is no exception. There is simply no context read yet, so it takes the initial state $h_0$ (usually a zero vector) as its previous state. Every step runs through the same cell in exactly the same way.
The part worth noticing is that $h_3$ is not information about the third token alone. It is a state in which the preceding context has accumulated in order, flowing $h_1 \rightarrow h_2 \rightarrow h_3$.
The information bottleneck: a single vector handed to the decoder
Once the encoder finishes, the decoder starts producing output. In vanilla Seq2Seq the decoder does not see $h_1, h_2, h_3$ at all. It receives only the last hidden state. This is called the context vector, $c$.
\[c = h_T\]Starting from that one vector, the decoder keeps producing output until the sentence ends, depending on the same $c$ at every step.
\[S_k = f(S_{k-1},\; \text{emb}(y_{k-1}),\; c)\]The limits of vanilla Seq2Seq
The architecture is simple and easy to implement, but problems appear as sentences get longer.
Information bottleneck: a single fixed-size vector $c$ has to carry the information of the entire input sentence. The longer the sentence, the more the early information is diluted on its way through many steps to $h_T$. An LSTM’s forget gate softens this somewhat, but the structural limit remains.
Fixed reference: whether the decoder is generating “Jane” or “to the store”, the information it consults is always the same $c$. That clashes with the intuition that each output step should focus on a different part of the input.
Summary
| Vanilla Seq2Seq | |
|---|---|
| Encoder output | a single $h_T$ |
| Context vector | fixed ($c = h_T$) |
| Decoder reference | always the same $c$ |
| Long-sentence performance | degrades sharply |
The core point is this.
Vanilla Seq2Seq accumulates the input context in order and depends on a single final state.
Coming up next
The next part looks at how to fix this bottleneck: letting the decoder see all of $h_1$ through $h_n$ and reference whichever states it needs, directly, at every step.