intorduce AdaDoRA implementation using both DoRA and AdaLoRa together #2969

OrMullerHahitti · 2025-12-25T13:16:27Z

This pull request introduces support for AdaDoRA (AdaLoRA with DoRA) to the codebase, enabling the use of DoRA (Weight-Decomposed Low-Rank Adaptation) within the AdaLoRA framework. The main changes add new modules and configuration options to support this, update the initialization and management of adapter layers, and provide a new GLUE finetuning script that demonstrates AdaLoRA/DoRA usage.

Key changes include:

AdaDoRA (DoRA for AdaLoRA) Support

Added the AdaDoraLinearLayer class in adalora/dora.py, implementing DoRA's magnitude vector and forward logic for AdaLoRA's SVD decomposition, including proper handling of singular values and rank pruning.
Updated adalora/__init__.py to export AdaDoraLinearLayer and AdaDoraLinearVariant for external use.
Modified adalora/config.py to allow use_dora (DoRA) in AdaLoRA by removing the previous error, enabling AdaDoRA configuration.

Adapter Layer and Variant Management

Extended AdaLoraLayer in adalora/layer.py to:
- Track lora_magnitude_vector in adapter_layer_names for DoRA support.
- Add a lora_variant dictionary to manage per-adapter variants (e.g., AdaDoRA).
- Allow initialization of adapter layers with the use_dora flag and resolve the correct variant, including initialization of the AdaDoRA variant if requested.

Utilities and Imports

Imported LoraVariant into adalora/layer.py to support adapter variants.

Example and Usage

Added a new script scripts/run_glue.py that demonstrates finetuning for GLUE tasks with AdaLoRA and AdaDoRA, including argument parsing for DoRA options, correct PEFT configuration, and integration of the new AdaLoRA/DoRA callback.

These changes collectively enable AdaLoRA models to leverage DoRA-style adaptation,while using extending dora.py implementation in LoRA module, with full support in both the core library and example scripts.

Adds weight-decomposed adaptation to AdaLoRA, introducing a learnable magnitude vector and DoRA-scaled forward/merge paths. Propagates configuration to layers and model, updates rank allocation to skip magnitude params, and recalculates magnitudes after pruning/masking. Introduces a GLUE finetuning script that wires AdaLoRA/DoRA, computes total steps for scheduling, and updates rank allocation via a training callback. Improves flexibility by supporting DoRA in AdaLoRA, safer merging, and clearer training integration; unmerging for DoRA remains unsupported.

- Added AdaDoraLinearLayer for handling AdaLoRA's 3-factor decomposition. - Introduced AdaDoraLinearVariant for managing variant-specific operations. - Updated AdaLoraLayer to support DoRA integration and manage variants. - Enhanced SVDLinear to utilize the new variant pattern for merging and unmerging. - Implemented tests for AdaDoraLinearLayer and AdaDoraLinearVariant to ensure functionality. - Updated AdaLoraModel to handle magnitude updates after rank pruning. - Added uv.lock file for dependency management.

…tation after rank pruning

BenjaminBossan

Thanks for opening a new PR to add DoRA support to AdaLoRA. I did a first review pass which focused on the general design of the implementation, I haven't done a detailed check of the DoRA logic or the example yet, as it makes more sense to finalize the integration first. Please check my comments.

I also saw that there are unrelated changes in the PR, like changing capitalization of comments or adding the uv lock file. Could you please revert all those changes? Finally, please run make style to ensure the linter is happy.

Moreover, before the PR can be merged, the docs should be updated, but we can leave this for later.

examples/adalora_with_dora/run_glue.py

src/peft/tuners/adalora/__init__.py

src/peft/tuners/adalora/config.py

src/peft/tuners/adalora/dora.py

src/peft/tuners/adalora/layer.py

BenjaminBossan · 2026-01-06T13:12:01Z

src/peft/tuners/adalora/layer.py

                    self.ipt[n] = torch.zeros_like(p)
                    self.exp_avg_ipt[n] = torch.zeros_like(p)
                    self.exp_avg_unc[n] = torch.zeros_like(p)
+                # skip update if no gradient (can happen at first step or for frozen params)


Is this something new that arose with AdaDoRA?

yes! This isn't strictly new to AdaDoRA but it rose while trainning on various tasks. it's actually a defensive fix I added while working on the integration.

The issue can occur in vanilla AdaLoRA too: at the very first training step, gradients haven't been computed yet, so p.grad is None. Calling
(p * p.grad).abs()
would then crash.

That said, I noticed it specifically while testing AdaDoRA because the magnitude vectors added another edge case to watch for. I figured a simple None check here is cheap insurance and makes the code more robust overall.

Happy to add a more detailed comment if you think it would help clarify!

Thanks for the details. Yes, if you could provide a small reproducer for this error, ideally with normal AdaLoRA, that would be great.

src/peft/tuners/adalora/model.py

src/peft/tuners/adalora/variants.py

src/peft/tuners/adalora/model.py

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

- Merge lora_magnitude_vector check with main conditional (with parentheses) - Remove internal exports (AdaDoraLinearLayer, AdaDoraLinearVariant) - Remove obsolete test_adalora_use_dora_raises test - Move run_glue.py to examples/adalora_adadora/ with README

…sted)

The method was a no-op (just pass) since resetting magnitude after pruning disabled DoRA by making the ratio m/||W+ΔW|| = 1. Removed the dead code for cleanliness.

Per reviewer feedback: variant resolution now happens inside the base class after _move_adapter_to_device_of_base_layer, ensuring correct ordering. SVDLinear just overrides resolve_lora_variant() to return the AdaDoRA variant.

Per reviewer feedback: consistent with LoRA DoRA implementation. Added _compute_lora_weight helper to compute B @ (A * E) * scaling / ranknum externally instead of inside get_weight_norm.

BenjaminBossan

Thanks for the updates. I did another review and still found a few areas for improvement, please check my comments.

uv.lock

BenjaminBossan · 2026-01-07T10:35:24Z

src/peft/tuners/adalora/dora.py

+        mag_norm_scale = (magnitude / weight_norm).view(1, -1)
+
+        # compute adapter output: x @ (A*E).T @ B.T * scaling / ranknum
+        lora_result = (x @ (lora_A * lora_E).T @ lora_B.T) * (scaling / (ranknum + 1e-5))


Ah, I forgot about the nn.Parameter thing, in that case just disregard what I said. This may cause issues with FSDP but it's a bit of an edge case.

Something else: For calculating the lora_result, can we not re-use the lora_weight that was already computed above?

BenjaminBossan · 2026-01-07T10:36:58Z

src/peft/tuners/adalora/layer.py

+        "lora_E",
+        "lora_embedding_A",
+        "lora_embedding_B",
+        "lora_magnitude_vector",


Let's remove lora_magnitude_vector here, since normal AdaLoRA doesn't have this. Instead, the DoRA variant should add it during its init. See how it's done in LoRA:

peft/src/peft/tuners/lora/variants.py

Lines 135 to 138 in 261366d

def init(module: Linear, adapter_name: str, **kwargs: Any) -> None:

if not module.lora_magnitude_vector:

# first dora layer being added, add lora_magnitude_vector to the list of learnable parameters

module.adapter_layer_names = module.adapter_layer_names[:] + ("lora_magnitude_vector",)

BenjaminBossan · 2026-01-07T10:37:18Z

src/peft/tuners/adalora/layer.py

        self.lora_dropout[adapter_name] = lora_dropout_layer
-        # Actual trainable parameters
-        # Right singular vectors
+        # actual trainable parameters


Please undo these kinds of changes. There are more of those below.

BenjaminBossan · 2026-01-07T10:37:48Z

src/peft/tuners/adalora/layer.py

        self._move_adapter_to_device_of_base_layer(adapter_name)
+
+        # initialize variant if needed (e.g., DoRA) - must happen after device move
+        if use_dora:


This check can be removed. if lora_variant is not None is enough.

BenjaminBossan · 2026-01-07T10:47:05Z

src/peft/tuners/adalora/layer.py

                    self.ipt[n] = torch.zeros_like(p)
                    self.exp_avg_ipt[n] = torch.zeros_like(p)
                    self.exp_avg_unc[n] = torch.zeros_like(p)
+                # skip update if no gradient (can happen at first step or for frozen params)


Thanks for the details. Yes, if you could provide a small reproducer for this error, ideally with normal AdaLoRA, that would be great.

BenjaminBossan · 2026-01-07T10:51:34Z

tests/test_adalora_dora.py

Thanks for adding tests, but let's use the existing test harness instead of introducing completely new tests. As a start, we can add AdaDoRA to the decoder model test matrix, check how it's done here:

peft/tests/test_decoder_models.py

Lines 88 to 94 in 261366d

AdaLoraConfig,

{

"task_type": "CAUSAL_LM",

"target_modules": None,

"total_step": 1,

},

),

After adding an entry for AdaDoRA, please ensure that the corresponding tests pass (something like pytest tests/test_decoder_models.py -k "adalora" should do the trick).

As for the tests in this file, I think they are all covered by existing tests and can thus be removed. You can take a look at the printed test coverage report to see if there are gaps in the tests when it comes to the newly added code.

examples/adalora_with_dora/run_glue.py

BenjaminBossan · 2026-01-07T10:53:23Z

examples/adalora_with_dora/README.md

@@ -0,0 +1,34 @@
+# AdaDoRA (AdaLoRA + DoRA) Example
+
+This example demonstrates how to use **AdaDoRA** - a combination of AdaLoRA's adaptive rank allocation with DoRA's weight-decomposition approach for parameter-efficient fine-tuning.


Let's link the papers.

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

OrMullerHahitti added 3 commits December 25, 2025 14:28

REF: Update AdaDoraLinearLayer to allow gradient-based magnitude adap…

8f83cef

…tation after rank pruning

BenjaminBossan requested changes Jan 6, 2026

View reviewed changes

OrMullerHahitti and others added 9 commits January 6, 2026 17:40

Update src/peft/tuners/adalora/dora.py

79ca71e

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

Update src/peft/tuners/adalora/variants.py

74d35c0

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

style: apply formatting and remove unnecessary comment

788d84c

Remove dora_init as it is from a self expiriment (could add if intere…

b3cda10

…sted)

change name of example dir

e285485

refactor: remove unused update_magnitude_after_pruning method

2bc5b65

The method was a no-op (just pass) since resetting magnitude after pruning disabled DoRA by making the ratio m/||W+ΔW|| = 1. Removed the dead code for cleanliness.

refactor: get_weight_norm now receives pre-computed lora_weight

c596f7b

Per reviewer feedback: consistent with LoRA DoRA implementation. Added _compute_lora_weight helper to compute B @ (A * E) * scaling / ranknum externally instead of inside get_weight_norm.

OrMullerHahitti requested a review from BenjaminBossan January 6, 2026 18:48

BenjaminBossan requested changes Jan 7, 2026

View reviewed changes

OrMullerHahitti and others added 2 commits January 7, 2026 13:04

Apply suggestion from @BenjaminBossan

33dffb0

Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com>

Delete uv.lock

49c0202

	def init(module: Linear, adapter_name: str, **kwargs: Any) -> None:
	if not module.lora_magnitude_vector:
	# first dora layer being added, add lora_magnitude_vector to the list of learnable parameters
	module.adapter_layer_names = module.adapter_layer_names[:] + ("lora_magnitude_vector",)

	AdaLoraConfig,
	{
	"task_type": "CAUSAL_LM",
	"target_modules": None,
	"total_step": 1,
	},
	),

		@@ -0,0 +1,34 @@
		# AdaDoRA (AdaLoRA + DoRA) Example

		This example demonstrates how to use AdaDoRA - a combination of AdaLoRA's adaptive rank allocation with DoRA's weight-decomposition approach for parameter-efficient fine-tuning.

intorduce AdaDoRA implementation using both DoRA and AdaLoRa together #2969

Are you sure you want to change the base?

intorduce AdaDoRA implementation using both DoRA and AdaLoRa together #2969

Uh oh!

Conversation

OrMullerHahitti commented Dec 25, 2025

AdaDoRA (DoRA for AdaLoRA) Support

Adapter Layer and Variant Management

Utilities and Imports

Example and Usage

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

BenjaminBossan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants