bahdanau-attention

an archive of posts with this tag

From Seq2Seq to Transformer: from accumulated context to direct reference · Part 3

The decoder's remaining problem - output context still accumulates

Attention solved the encoder-side information bottleneck. But the decoder itself is still a sequential structure that depends on its previous state. Here is why that becomes the final motivation for the Transformer.

attention seq2seq bahdanau-attention query-key-value nlp

deep-learning

2023-06-28 · 3 min read
From Seq2Seq to Transformer: from accumulated context to direct reference · Part 2

Attention Seq2Seq - don't just look at the last state, look directly

How do you fix the vanilla Seq2Seq bottleneck? By letting the decoder reference every encoder hidden state directly, "as needed".

attention seq2seq bahdanau-attention query-key-value nlp

deep-learning

2023-06-02 · 4 min read