Conformer misses relative pos encoding

[albertz]

There is self attention without any pos encoding.

Related is Transformer rel pos #74.

There probably should be a nn.RelPosSelfAttention, similar as nn.SelfAttention but with rel pos encoding. It should use the Transformer XL rel pos encoding. (We can separately test other rel pos encodings later.)

[albertz]

See here for an example: https://github.com/espnet/espnet/blob/4138010fb66ad27a43e8bee48a4932829a0847ae/espnet2/asr/encoder/conformer_encoder.py#L132
(Up-to-date 2022-10-17.)

RelPositionMultiHeadedAttention

RelPositionalEncoding

The RelPositionalEncoding is then also part of the frontend (e.g. Conv2dSubsampling6) (also see #219).
Specifically, at the end of the frontend, there is another Linear to the model dim, and then the pos enc (here).

RelPositionalEncoding must be used together with RelPositionMultiHeadedAttention.

[albertz]

Note that all our existing Conformer recipes in RETURNN currently use the RETURNN RelativePositionalEncodingLayer (example). I.e. that is not the Transformer XL variant, as it is standard now. We probably should directly implement the Transformer XL variant now.

But for direct comparisons to old setups, we might also need to implement the old variant.

[albertz]

Interestingly, the Fairseq RelPositionalEncoding looks almost identical to the ESPnet implementation? It looks like the code was copied. But there is no mention at all about this. Fairseq is MIT, ESPnet is Apache licence. (@sravyapopuri388 @sw005320 @pengchengguo maybe you know the history?)

[albertz]

For reference, the original Transformer-XL rel_multihead_attn implementation.

[pengchengguo]

Hi @albertz,

I think the Fairseq RelPositionalEncoding is based on ESPnet as they distinguish different RPE/Attention types between Fairseq and ESPnet here: https://github.com/facebookresearch/fairseq/blob/b7b7928065ec90bef8f9a489b0128a4dce560d57/fairseq/modules/conformer_layer.py#L187.

[sravyapopuri388]

@albertz The fairseq implementation is based on the Espnet one for relative positional encoding. Seems like we are indeed missing some citations. Will make sure to add them soon. Thanks!