[Question] What is the main difference between "AutoModelForCasualLM" and "PeftModelForCausalLM"?

[daehuikim]

I also wrote it down in peft repo. However this issue is also related to transformers. So i write my question here again.
issue is here in peft(huggingface/peft#1245)

Hello, Sorry for naive question.
I noticed that themodel.generate() function performed differently when inferrence right after train with trainer.model and after merge and unload. (Every params are the same.)
So I checked two different object with simple print function.
Difference was the object that contains model.

1. model = trainer.model

PeftModelForCausalLM(
  (base_model): LoraModel(
    (model): LlamaForCausalLM(
      (model): LlamaModel(
        (embed_tokens): ModulesToSaveWrapper(
          (original_module): Embedding(32008, 5120)
          (modules_to_save): ModuleDict(
            (default): Embedding(32008, 5120)
          )
        )
        (layers): ModuleList(
          (0-39): 40 x LlamaDecoderLayer(
            (self_attn): LlamaAttention(
              (q_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (k_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (v_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (o_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=5120, bias=False)
              )
              (rotary_emb): LlamaRotaryEmbedding()
            )
            (mlp): LlamaMLP(
              (gate_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=13824, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=13824, bias=False)
              )
              (up_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=5120, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=13824, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=5120, out_features=13824, bias=False)
              )
              (down_proj): Linear4bit(
                (lora_dropout): ModuleDict(
                  (default): Dropout(p=0.1, inplace=False)
                )
                (lora_A): ModuleDict(
                  (default): Linear(in_features=13824, out_features=64, bias=False)
                )
                (lora_B): ModuleDict(
                  (default): Linear(in_features=64, out_features=5120, bias=False)
                )
                (lora_embedding_A): ParameterDict()
                (lora_embedding_B): ParameterDict()
                (base_layer): Linear4bit(in_features=13824, out_features=5120, bias=False)
              )
              (act_fn): SiLUActivation()
            )
            (input_layernorm): LlamaRMSNorm()
            (post_attention_layernorm): LlamaRMSNorm()
          )
        )
        (norm): LlamaRMSNorm()
      )
      (lm_head): ModulesToSaveWrapper(
        (original_module): Linear(in_features=5120, out_features=32008, bias=False)
        (modules_to_save): ModuleDict(
          (default): Linear(in_features=5120, out_features=32008, bias=False)
        )
      )
    )
  )
)

2. AutoModelForCasualLM.from_pretrained(after merging lora adapter)

LlamaForCausalLM(
  (model): LlamaModel(
    (embed_tokens): Embedding(32008, 5120)
    (layers): ModuleList(
      (0-39): 40 x LlamaDecoderLayer(
        (self_attn): LlamaAttention(
          (q_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (k_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (v_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (o_proj): Linear4bit(in_features=5120, out_features=5120, bias=False)
          (rotary_emb): LlamaRotaryEmbedding()
        )
        (mlp): LlamaMLP(
          (gate_proj): Linear4bit(in_features=5120, out_features=13824, bias=False)
          (up_proj): Linear4bit(in_features=5120, out_features=13824, bias=False)
          (down_proj): Linear4bit(in_features=13824, out_features=5120, bias=False)
          (act_fn): SiLUActivation()
        )
        (input_layernorm): LlamaRMSNorm()
        (post_attention_layernorm): LlamaRMSNorm()
      )
    )
    (norm): LlamaRMSNorm()
  )
  (lm_head): Linear(in_features=5120, out_features=32008, bias=False)
)

I think both modes should work exactly the same way, but when I inferred with the model.generate function, I found that #1 (PeftModelForCausalLM) works much more accurately. I'd like to know why, is there a theoretical or engineering reason for this?
Thanks for watching my long long question!

[ArthurZucker]

cc @younesbelkada

[DopeorNope-Lee]

Hmm looks interesting. I think, when you print it out, first thing is considering original+ adapter and last thing is considering only structure of it’s backbone.
Did you considered a random state, before inference?

Sometimes, same input’s result can be different when you inference it.

[daehuikim]

Hmm looks interesting. I think, when you print it out, first thing is considering original+ adapter and last thing is considering only structure of it’s backbone. Did you considered a random state, before inference?

Sometimes, same input’s result can be different when you inference it.

The reason why the structure of the last one feels exactly the same as the structure of the backbone model is because the lora adapter of this one has already been merged, so there is no numerical difference.
The only difference is the Object they are carried (PeftModelForCausalLM, AutoModelForCasualLM).

And 2 models are initialized from different sources. one from PeftModelForCausalLM.from_pretrained and one for from AutoModelForCasualLM.from_pretrained. However, every params, configuration files are the same except the object.

[younesbelkada]

Hi @daehuikim
Thanks for your issue, recall the formulation of the LoRA adapters in the figure below:

The fundamental difference between AutoModelForCausalLM and PeftModelForCausalLM, in your case is that the AutoModelForCausalLM contains the merged adapter whereas the second model contains the model with LoRA attached on them. Technically the inference results should be the same as the LoRA operations can be simply rewritten as a refactorization of simple matrix multiplication:

Now regarding your issue, it seems you have further trained your lm_head as well. I think merge_and_unload does not properly take care of that, but I am not sure. Can you confirm which PEFT version are you using now?

[daehuikim]

Hi @daehuikim Thanks for your issue, recall the formulation of the LoRA adapters in the figure below:

The fundamental difference between AutoModelForCausalLM and PeftModelForCausalLM, in your case is that the AutoModelForCausalLM contains the merged adapter whereas the second model contains the model with LoRA attached on them. Technically the inference results should be the same as the LoRA operations can be simply rewritten as a refactorization of simple matrix multiplication:

Now regarding your issue, it seems you have further trained your lm_head as well. I think merge_and_unload does not properly take care of that, but I am not sure. Can you confirm which PEFT version are you using now?

Thanks for describing the issue correctly.
My peftconfig for training is like below: (lm_head is not trained by trainer)

peft_config = LoraConfig(
        lora_alpha=lora_alpha,
        lora_dropout=lora_dropout,
        r=lora_r,
        bias="none",
        task_type="CAUSAL_LM",
        target_modules= [
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        ],
        modules_to_save=[
            "embed_tokens",
            "lm_head"
        ]
    )

And this is my peft version

$ pip freeze | grep peft
peft==0.6.2

Thanks for answering! @younesbelkada

[younesbelkada]

Thanks @daehuikim
You are adding

        modules_to_save=[
            "embed_tokens",
            "lm_head"
        ]

in the lora config, this will make the lm head and the input embedding layer trainable, hence the strange behaviour you are facing (I am not sure we support merge and unload for models that have modules_to_save). Can you try without modules_to_save?

[daehuikim]

Thanks @daehuikim You are adding

        modules_to_save=[
            "embed_tokens",
            "lm_head"
        ]

in the lora config, this will make the lm head and the input embedding layer trainable, hence the strange behaviour you are facing (I am not sure we support merge and unload for models that have modules_to_save). Can you try without modules_to_save?

Thanks again @younesbelkada

modules_to_save wrapper save the modules not to be trained from trainer.
Also, embed_tokens, lm_head must to be saved when there exist added_tokens.
Therefore, modules_to_save block is necessary for my task.
I will see what happens in merge_and_unload for finding this out.
Thank you again!

[daehuikim]

https://github.com/huggingface/peft/blob/21c304f6f6ea62c7dcbef8e201d178a7575b471d/src/peft/tuners/lora/model.py#L419

elif isinstance(target, ModulesToSaveWrapper):
                # save any additional trainable modules part of `modules_to_save`
                setattr(parent, target_name, target.modules_to_save[target.active_adapter])

@younesbelkada
modules_to_save preserve modules from parent model.
Theoretically both model should reproduce same outputs.
But why this still happen? I am curious.

[SatireY]

Hi, I'm having the same problem as you, in my scenario I also need to add some tokens for training and I've also set the

modules_to_save=[
            "embed_tokens",
            "lm_head"
      ]

May I ask if the problem is solved now? How did you end up solving it, looking forward to your reply.

[daehuikim]

Hi, I'm having the same problem as you, in my scenario I also need to add some tokens for training and I've also set the

modules_to_save=[
            "embed_tokens",
            "lm_head"
      ]

May I ask if the problem is solved now? How did you end up solving it, looking forward to your reply.

I have no idea yet. whole codes seems same in peft and transformers. I guess it could be some weight initialization problem

[SatireY]

Hi, I'm having the same problem as you, in my scenario I also need to add some tokens for training and I've also set the

modules_to_save=[
            "embed_tokens",
            "lm_head"
      ]

May I ask if the problem is solved now? How did you end up solving it, looking forward to your reply.

I have no idea yet. whole codes seems same in peft and transformers. I guess it could be some weight initialization problem

Roger that. Thank you for your reply.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[younesbelkada]

Hi there! Is the issue solved? Let me know if you need more clarification