Training difficulties

[LouisSerrano]

Hi, I am experiencing some difficulties during the training of magvit2. I don't know if I made some mistakes somewhere or where the problem might be coming from.

It seems that my understanding of the paper might me be erroneous, I tried with 2 codebooks of size 512 and I can't seem to fit the training data. The training is really unstable. I tried to replace the LFQ with a classical VQ and it was more stable and was able to converge.
What is the config that you tried for training the model ?

[lucidrains]

@LouisSerrano hey Louis and thank you for reporting this

that is disappointing to hear, as I had high hopes for LFQ. there is the possibility that I implemented it incorrectly, but the authors had already given it a code review, so that seems less likely. i'll be training the model tonight (regaining access to my deep learning rig) so i can see this instability for myself.

[lucidrains]

@LouisSerrano as a silver lining, i guess your experiments show the rest of the framework to be working ok. i can run a few toy experiments on LFQ this morning before running it on a big image dataset tonight and see what the issue is

[lucidrains]

@LouisSerrano just to make sure, are you on the latest version of the vector-quantize-pytorch library?

[LouisSerrano]

Ok thank you very much, it might simply be an error from my side, probably in the configuration. I used the model config that you suggested in the Readme.md, and for LFQ I used 1 codebook with size 512.

[LouisSerrano]

@lucidrains yes I am using 1.10.4

[lucidrains]

@LouisSerrano no that should be fine, you could try increasing to 2048 or 4096, but shouldn't make a big difference

[LouisSerrano]

@lucidrains As a precision I tried with a different dataset, a smaller one actually, but which should be less challenging than the ones from the paper.

[lucidrains]

@LouisSerrano this is kind of a stab in the dark, but would you like to try adding lfq_activation = nn.Tanh(), # from torch import nn to your VideoTokenizer init? (v0.0.64)

[lucidrains]

they didn't have this in the paper, but i think it should make sense

[lucidrains]

@LouisSerrano let me do some toy experiments right now and make sure it isn't some obvious bug

[LouisSerrano]

@lucidrains Ok I am going to try to increase the codebook size, just in case. Sure, I can check with tanh activation.

[lucidrains]

@LouisSerrano thank you! 🙏

[LouisSerrano]

@lucidrains Thanks for the awesome work !

[lucidrains]

@LouisSerrano yea no problem! well, we'll see if this work pans out. thank you for attempting to replicate!

[lucidrains]