fix 'intermediate_size' in Llama configuration files after the 'mlp_type' option was removed

[tiandeyu-cs]

After the 'mlp_type' option was removed, the regular-type and the llama-type of MLP share the same implementation, and the "mlp_type" is now specified by whether the activation is gated or not.
However, this changes the meaning of the 'intermediate_size' option in llama configuration files. The code (megatron/model/transformer.py) now treats 'intermediate_size' as the size of the output tensor of the first linear layer in the MLP, which includes the first layer and the gated layer of llama-type MLP. This means that the code actually halves the 'intermediate_size' of llama-type MLP. Meanwhile, the code multiplies the 'intermediate_size' by (2/3), which means the actual 'intermediate_size' is only (1/3) of the intended size in the configuration file.

To fix this problem, I revised the llama configuation files, and set the 'intermediate_size' to 3 times as its intended value.

[CLAassistant]

All committers have signed the CLA.

[aflah02]

Hi @tiandeyu-cs
I just ran into this issue but why is the modified intermediate dim for 7B Llama2 model 32768, shouldn't it be 3x11008 = 33024?
HF and LitGPT both use 11008 - https://huggingface.co/meta-llama/Llama-2-7b-hf/blob/main/config.json#L11 & https://github.com/Lightning-AI/litgpt/blob/a5021be4bb48e27779586b56b062a1749ecb232f/litgpt/config.py#L1474

[aflah02]

Also I used the 32768 value while training in the config but during export I got a size mismatch error and set the intermediate dim value in config as 11008 (for it to export)
Error when I use same config -

[aflah02]

It would be great if you can share what's the formula to use to get the intermediate size for the config since it is not 3x in the above case but more like 2.97x. This will help me make my other config files for smaller models.

[tiandeyu-cs]

The code linked here calculates 1/3 of the intermediate_size and pads it to a multiple of the multiple_of variable (which comes from the mlp_multiple_of parameters from the configuration file, retrieved by the the code linked here).

Therefore, if we set intermediate_size as 32768 and mlp_multiple_of as 256, then the code first calculates ffn_dim_in as (32768 // 3 = 10922), then the code pads it to a multiple of 256, which should be 11088. I think you can directly set the intermediate_size as (11088 * 3 = 33024) and it will not cause errors in the code. I chose to use 32768 in the configuration file because I thought it is more elegant :)

[aflah02]

Thanks @tiandeyu-cs
I had come to the same hypothesis but glad to get confirmation

[aflah02]

p.s. it would be nice to have this as a comment in the yaml file for future wanderers or just maybe making it 11008*3 would save some people from confusion