Attention Seq2Seq - don't just look at the last state, look directly

Recap

Part one came down to a single point. Vanilla Seq2Seq compresses all the information into the encoder’s last hidden state ($h_T$) and hands that to the decoder. The longer the sentence, the more the early information is diluted, and the decoder always looks at the same single vector.

The question this time is simple. Why should we only look at the last state?

The idea behind attention

Vanilla Seq2Seq depended on a single $c = h_T$. Attention changes that. It opens up all of $h_1, h_2, \ldots, h_n$ to the decoder, and lets the decoder decide dynamically — based on its current state $S_k$ — which $h_i$ to weigh more heavily.

h1, h2, h3
↑   ↑   ↑
|___|___|
    ↑
Decoder (S_k)
    ↓
c_k = weighted sum (different at every step)

The key is that the context vector is no longer fixed. Every time the decoder state changes, $c_k$ changes with it.

Reading attention as query, key, value

The variables in attention map exactly onto $Q, K, V$.

Concept	What it actually is in attention Seq2Seq	Role
Query ($Q$)	the decoder’s current state $S_k$	“which input should I be looking at right now?”
Key ($K$)	the encoder hidden state $h_i$	“let me tell you what kind of information I am”
Value ($V$)	the encoder hidden state $h_i$	“if I’m relevant, take my actual content”

What matters here is that $K$ and $V$ are both the same $h_i$. That fact is the decisive link that later leads to self-attention. We will come back to it.

The steps of the attention computation

Step 1: relevance scores

Compare the current decoder state ($Q = S_k$) with each encoder state ($K = h_i$) via an inner product.

\[\text{score}(Q, K_i) = \langle S_k,\, h_i \rangle\]

A larger inner product means “$h_i$ is highly relevant to the current decoder state”.

Step 2: turning scores into weights

Convert the scores into a probability distribution that sums to one.

\[w_{ki} = \text{softmax}(\text{score}(Q, K_i)) = \frac{\exp(\langle S_k, h_i \rangle)}{\sum_j \exp(\langle S_k, h_j \rangle)}\]

Step 3: building a dynamic context vector

Multiply the weights by $V_i\,(= h_i)$ and sum.

\[c_k = \sum_i w_{ki} \cdot h_i\]

Rolling the three steps into one gives the full attention formula.

\[c_k = \sum_i \text{softmax}(\langle S_k, h_i \rangle) \cdot h_i\]

What concretely changes

Say we are translating “Jane went to the store”. The encoder reads each token and produces $h_1(\text{Jane}),\; h_2(\text{went}),\; h_3(\text{to the}),\; h_4(\text{store})$.

When generating the word for “Jane”, $Q = S_1$ shows high similarity with $h_1$ (“Jane”). The result is roughly $w \approx [0.90,\, 0.04,\, 0.03,\, 0.03]$, so $c_1 \approx h_1$.

When generating the words for “to the store”, the decoder state changes to $S_2$ and similarity with $h_4$ (“store”) rises. Now $w \approx [0.02,\, 0.03,\, 0.03,\, 0.92]$, so $c_2 \approx h_4$.

In vanilla Seq2Seq both steps used the same $c = h_T$. Attention computes a different $c_k$ at every step.

What attention fixed, and what remains

Attention solved the encoder-side problem. The bottleneck at a single final state is gone, and the decoder can directly reference whichever input position it needs at each step.

But the decoder is still a sequential structure that depends on its previous state.

\[S_k = f(S_{k-1},\; \text{emb}(y_{k-1}),\; c_k)\]

$c_k$ is now computed dynamically by attention, but producing $S_k$ still requires $S_{k-1}$. The output-side context is still passed along, accumulated in order.

That is the problem for the next part.

Summary

	Vanilla Seq2Seq	Attention Seq2Seq
Encoder reference	a single $h_T$	all of $h_1 \sim h_n$
Context vector	fixed	dynamic (different at every step)
Q, K, V	not applicable	$Q=S_k,\; K=h_i,\; V=h_i$
Encoder bottleneck	yes	no
Decoder sequentiality	yes	still yes

The core point is this.

Attention does not depend on a single final state; it directly references whichever encoder states it needs, every time.

Coming up next

Attention solved access to encoder information, but the decoder itself is still sequential. The next part looks at why that remaining problem becomes the motivation for the Transformer.