WeightLoRA: Keep Only Necessary Adapters

Andrey Veprikov, Vladimir Solodkin, Alexander Zyl, Andrey Savchenko, Aleksandr Beznosikov

WeightLoRA keeps only the adapters that matter. It augments LoRA with a short warm-up stage that learns adapter importance, selects the most critical updates, and continues fine-tuning on this compact subset.

WeightLoRA+ extends this idea to the fixed-budget setting: it keeps the total LoRA parameter budget unchanged, but reallocates rank toward the selected adapters, leading to stronger accuracy under the same budget.

Experiments

The paper evaluates:

• GLUE with DeBERTaV3-base and Llama 3
• MathQA, GSM8K, HellaSwag, BoolQ and ARC-Challenge with Qwen3-8B
• SQuAD v1.1 / v2.0, XSum and CNN/DailyMail with DeBERTaV3-base
• GSM8K with Llama-3.1 70B

Llama-3.1 70B on GSM8K

The main experimental results are reported in the paper. See the tables there for the full comparison across tasks, models, and ranks. Here we highlight one of the largest-scale examples: on Llama-3.1 70B with GSM8K.

WeightLoRA+ improves over standard LoRA at every tested rank, while WeightLoRA stays competitive with a substantially smaller number of trainable parameters.

Ablation Studies

The paper contains multiple ablations. We show here the two most practical ones: memory and training time. They illustrate that disabling unnecessary adapters gives a clear memory benefit, while the selection stage does not introduce a noticeable wall-clock overhead.

Repository Structure

LIB/
├── src/         # core training code, configs, trainers, and task-specific pipelines
├── scripts/     # runnable experiment scripts for GLUE and Qwen
├── peft/        # local PEFT fork with the WeightLoRA integration
├── assets/      # README figures
└── notebooks/   # analysis and visualization notebooks

Setup

python3.11 -m venv wlora_venv
source wlora_venv/bin/activate
pip install -r requirements.txt

Running Experiments

Example entry points:

# Qwen3-8B + WeightLoRA on BoolQ by default
bash scripts/llm/qwen_weight_lora.sh

# Qwen3-8B + WeightLoRA+ / fixed-budget rank reallocation
bash scripts/llm/qwen_fat_lora.sh

# DeBERTaV3-base + WeightLoRA on GLUE
bash scripts/glue/deberta/weight_lora.sh

The unified experiment entry point is src/run_experiment.py.

Note: some launch scripts contain machine-specific defaults such as CUDA_VISIBLE_DEVICES, short debug schedules, and wandb logging flags. Adjust them before launching full-scale runs.