On padding the input with -1

[hquan7395]

The comment at

event-ssm/event_ssm/dataloading.py

Line 89 in a127667

# pad tokens with -1, which results in a zero vector with embedding look-ups

says that padding -1 corresponds to zero in the embedding layer. But when I check the trained params with

state['params']['encoder']['encoder']['embedding'][-1]

I got

Array([-0.06191076, 0.18668179, -0.25837645, 0.1488393 , 0.11525428, 0.18183647, -0.30500978, 0.07464553, -0.3888737 , 0.10938968, -0.02655256, -0.349307 , -0.03291983, -0.040106 , 0.2369387 , -0.01890478, -0.21027383, -0.00453783, -0.1900245 , 0.01763803, -0.09983198, -0.3064041 , -0.14400026, -0.05854894, 0.10559192, 0.04855605, 0.1330904 , 0.00867153, -0.04014621, 0.23416986, 0.14465654, 0.3036504 ], dtype=float32)

Is this not intended?

[markschoene]

Flax linen nn.Embed returns a zero vector for negative indices. However, explicitly querying [-1] on the tensor will return the last entry of the tensor sliced along the corresponding dimension (typical Python behavior). I’d suggest to test by calling the embedding object directly instead of the state dict. If this still yields non zero tensors, use the version indicated in the README.md

In any case, if you use the classification model, there will be a mask applied to the loss that removes all the terms that were tokenized as -1 anyways. The above steps should help even if you want to use the model without masking.

[markschoene]

Could you let me know if this addresses your concerns? I’d be happy to answer more questions if there are any in the process.

[hquan7395]

Thanks for your answer! Another naive question: are these embeddings trainable, or do they stay fixed?

[markschoene]

The embeddings are trainable. They actually make up most of the parameters. For example, 4 million parameters for embeddings in the DVS model versus just a few hundred K for the state-space model. For on-device inference, it would be much better to compute the embeddings rather than learning them, or at least to find a low rank representation. So if you want to continue research on this, I would suggest to try positional encoding similar to what transformers use these days, or to learn a low rank decomposition of the embedding matrix.