Integrate: LoRA vs Full Fine-tuning: An Illusion of Equivalence

[BenjaminBossan]

Feature request

In the paper LoRA vs Full Fine-tuning: An Illusion of Equivalence, the authors describe a method to detect the degree of forgetting caused by LoRA by identifying so called "intruder dimensions". They also describe a method to mitigate this, at the cost of possibly reducing performance on the fine-tuning task.

For PEFT, it could be interesting to add a new function that implements this mitigation on a fine-tuned LoRA model (and possibly for PEFT methods). This issue is for discussing this paper and the possible applications to PEFT. It is also a call for contributions -- if you are interested in creating a PR, please post here before starting the work to avoid duplicate effort.

Design

The proposed mitigation is to identify the intruder dimensions on $\Delta W$ and scale them down. At the moment, it is unclear to me if, after downscaling, the $\Delta W$ can be projected back onto the original lora_A and lora_B. Depending on whether this is possible, I see two ways of handling this.

Back projection possible

In this case, we can add a new LoRA adapter that is a copy of the existing, fine-tuned one, apply the mitigation, then project back onto lora_A and lora_B of this adapter:

def reduce_intruder_dimension(
    peft_model, old_adapter_name="default", new_adpater_name="new_adapter", top_k=10, threshold_epsilon=0.5, mitigation_lambda=0.75,
):
    # check if peft method is supported
    peft_model.add_adapter(new_adpater_name, peft_model.peft_config[old_adapter_name])
    # copy LoRA weights to the new adapter
    # apply mitigation to new adapter
    peft_model.set_adapter(new_adpater_name)
    return peft_model

For the mitigation_lambda parameter, check section 5 of the paper. top_k is the number of singular vectors to check (could be float for top k%), epsilon is the similarity threshold below which we consider a vector to be an intruder (see original code).

Back projection not possible

If back projection cannot be done, I would suggest to merge and unload with the new weights, as unmerging would not be trivially possible (but could possibly be added later).

def merge_and_unload_with_reduced_intruder_dimensions(
    peft_model, adapter_name="default", top_k=10, threshold_epsilon=0.5, mitigation_lambda=0.75,
):
    # check if peft method is supported
    for layer in peft_model.named_parameters():
        if isinstance(layer, LoraLayer):
            delta_weight = layer.get_delta_weight(adapter_name)
            # apply mitigation
            layer.get_base_layer().weight += delta_weight
    return peft_model.merge_and_unload(adapter_names=adapter_name)

Out of scope

This feature does not include the testing code itself. This means, it would be up to the user to validate that the tradeoff between task performance and forgetting is to their liking (controllable via mitigation_lambda).

Reference

• Repository
• Paper

ping @reeceshuttle as the first author of the paper.

Your contribution

I can contribute with discussion, guidance, reviews.

[dhyaneesh]

Mind if I take a crack at this ?

[BenjaminBossan]

@githubnemo Did you already start with this?

[githubnemo]

I experimented with this a bit and there are some questions (e.g., reeceshuttle/intruder-dimensions#3). I think whatever implementation it takes, we'll need some experiments to back up that it actually does what it says. To me, that should be the focus: a rough implementation that implements the gist and a good example that shows the benefit of the method as implemented.

Thank you @ved1beta for pressing forward with an implementation but since it is not clear how this is even implemented or if it works well, I think that a full-blown implementation with all bells and whistles is a bit premature. We've also stated in the issue that we'd like to discuss this before going forward with an implementation.

@dhyaneesh if you feel this is up your alley, feel free to take a stab at implementing this.

[githubnemo]

Small update:

I have attempted a crude test regarding whether to use dW or W + dW and I think the answer is to take dW.

When using dW to compute the intruder dimensions and subtracting them with varying lambda values we get a moderate loss in task accuracy (expected) and an improvement in LM loss (i.e. we're getting closer to the generation of the base model, so less forgetting). While this is a very crude test since we only have one sample for the forgetting metric the contrast is quite stark.

When using W+dW, we're getting the opposite behavior of what we'd want. The accuracy falls off more drastically while the loss increases (more forgetting).

Note that I used threshold=0.9 for both cases, maybe that's a potential cause for issues and more tuning is needed.

You can see how this data was produced in the following notebooks:

• dW: https://gist.github.com/githubnemo/8a999361cb09d46cd4e41e01e68dd1a4
• merged: https://gist.github.com/githubnemo/aa1e406f6bdf5c2f116d5686a86cca90

Edit: I just noticed that the graphs above show a loss of < 10 for "Umodified LoRA" which is wrong - that's the base model loss.

[reeceshuttle]

@githubnemo thanks again for your work on this! For visibility, wanted to call attention to the answers to the implementation questions raised above(reeceshuttle/intruder-dimensions#3). I'm glad results appear to replicate.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.