Attention Mechanism: New Approach to Sequence Alignment
Dzmitry Bahdanau and colleagues propose attention mechanism, solving long-range dependency problem in sequence-to-sequence models.
Overview
In 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio published a groundbreaking paper introducing attention mechanism in neural machine translation. Traditional sequence-to-sequence models needed to compress entire input sequences into fixed-length vectors, causing translation quality degradation for long sentences. Attention mechanism allows the decoder to "attend" to different parts of the input sequence at each time step, dynamically selecting relevant information. This innovation significantly improved machine translation quality and laid the conceptual foundation for the later Transformer architecture.
Study Notes
Background, mechanics, impact, and key takeaways in one read
Background & Context
Before 2014, neural machine translation relied on an encoder–decoder design: an encoder compressed the whole source sentence into a single fixed-length vector from which a decoder generated the translation. Sutskever et al.'s 2014 seq2seq model reached 34.8 BLEU on WMT'14 English-to-French, but only by reversing the source order, and its quality dropped sharply with longer sentences—the fixed vector was an information bottleneck. In September 2014, Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio of Université de Montréal submitted Neural Machine Translation by Jointly Learning to Align and Translate (ICLR 2015), proposing the attention mechanism: instead of one compressed vector, the decoder softly consults relevant source positions when producing each target word. This idea became the direct conceptual ancestor of self-attention and the Transformer.
Key Points
- 1Attention lets the decoder compute a fresh context vector for every target word, instead of reusing one fixed vector—lifting the information bottleneck on long sentences.
- 2Alignment and translation are learned jointly inside a single differentiable network, removing the need for external word-aligners and phrase tables.
- 3It uses an additive scoring function: a small feed-forward network scores the decoder state against each source annotation before a softmax turns scores into weights.
- 4The encoder is a bidirectional RNN, so each source annotation blends both past and future context for that position.
- 5On long sentences it clearly beats attention-free seq2seq and approaches the quality of phrase-based statistical machine translation.
- 6Often called additive attention, it is the direct conceptual ancestor of self-attention and the Transformer architecture.
How It Works
In the classic seq2seq, the source is squeezed into one vector c. Bahdanau instead keeps the encoder's hidden states h_1…h_Tx as annotations. To produce target word i, the model scores every source position with e_ij = a(s_{i-1}, h_j), where a is a small trainable feed-forward network, then normalizes the scores with a softmax into weights α_ij, and forms the context vector c_i = Σ α_ij h_j. Intuitively, the model learns where to look in the source for each output word. Because scoring and weighting are differentiable (soft alignment), gradients flow straight into the alignment; the bidirectional encoder gives each h_j both left and right context. Unlike Luong's later multiplicative (dot-product) attention, Bahdanau's is additive attention.
Impact & Legacy
Attention quickly became a standard building block for sequence modeling and spread to image captioning, speech recognition, and reading comprehension. Its key legacy was promoting dynamically reading context from an RNN add-on into a standalone operation; three years later the Transformer removed recurrence entirely and made attention the protagonist. Limitations remained: it still rode on RNNs, its cost grew quadratically with sequence length, and additive scoring was slower than later dot-product attention—issues the Transformer's parallel design largely resolved.
Key Takeaways
- The fixed-length vector is an information bottleneck; long sentences suffer most.
- Attention is fundamentally about weighting inputs dynamically for the current step, task-agnostic.
- Soft, differentiable alignment lets alignment be trained end-to-end with translation.
- Big ideas often start as narrow fixes before becoming general-purpose paradigms.
Glossary
A mechanism that, when producing each output, dynamically focuses on relevant inputs via learned weights instead of a single compressed representation.
The scoring scheme proposed by Bahdanau, using a small feed-forward network to compute match scores between query and keys.
A differentiable alignment that assigns continuous weights over all input positions, letting gradients flow through alignment.