Numerator: exp(e_ij)

The passage around this formula

Bahdanau, Cho and Bengio proposed letting the decoder search the source for the parts relevant to each output word, rather than reading from a single compressed vector [ 9 ] . The decoder computes, at each output step i , a context vector as a weighted sum of all encoder states: $c_i = \sum_{j=1}^{T_x} \alpha_{ij} h_j, \qquad \alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}$. The capacity of the intermediate representation now grows with the input rather than being fixed in advance. They further reported that the learned alignments corresponded well with human linguistic intuition — an interpretability result that arrived free with a performance fix, which is rare.

Read this part in the article →