
Date: 2026-08-26
Suitable for readers: graduate students, new scientific research, engineering readers who want to make the parameters efficient fine-tuning
Search date: 2026-08-26
Abstract
LoRA and QLoRA are often described as ways to fine-tune large models with less GPU memory. Three details are easy to get wrong in a reproduction. Which weights are frozen? Does the low-rank update have the right shape? Which memory costs does 4-bit quantization reduce: base weights, gradients, or optimizer states? This walkthrough connects the LoRA equations and official code to a minimal PyTorch layer, a QLoRA memory estimate, and a troubleshooting guide. Reproducing experiments at the scale of Guanaco or GPT-3 is outside the scope of the local run.
This actual verification is limited by the fact that the current workspace is not installed with PyTorch: the matching script has completed the Python syntax check, and runs the parameter/video budget check that does not depend on PyTorch; Toy LoRA The training code needs to be installed and then manually run after the dependencies are installed. When the paper is involved in the paper, only the paper report or the official repository description is cited when it is involved, and the local TOY inspection is not extrapolated as the original paper to reproduce.
Contents
- Why do you reproduce LoRA first instead of running PEFT first?
- Formulas and Tensor Shapes of LoRA
- A guide to the official implementation
- QLoRA multi-provincial video memory, where is the province
- Minimum Experiment and Acceptance Agreement
- How to report reproducing experimental boundaries
- Failure checklist
- Questions for further study
- Summary and References
Why do you reproduce LoRA first instead of running PEFT first?
Parameters efficient Fine-Tuning (Parameter-Efficient Fine-Tuning, PEFT) refers to not updating all the parameters of the model, but only trains a small number of new parameters or a few original parameters. It is suitable for newcomers to scientific research, because the scale of the experiment can be much smaller; it is also easy to mislead people, because the library call is too simple and hides the key constraints. if just write get_peft_model(model, config), it is difficult to see what matrix the LoRA increment is added to, and it is difficult to judge whether the training parameter ratio is as expected.
The core hypothesis of the LoRA paper is: the weight update required for the downstream tasks Delta W It may have a low inner rank, so there is no need to fully learn an update in the same shape as the original matrix. For a linear layer, the original weight W0 Freeze, train only two small matrices A and B, use when forward W0 + Delta W, in Delta W = B A. In this way, the first acceptance standard for recurrent is very clear: the base weight must be requires_grad=False, only the LoRA parameter and the necessary task headers participate in the optimization.
This is different from "only freezing a part of the layer". LoRA still lets the input pass the same computation path of the original model, just a low rank increment in parallel next to the target linear layer. at Inference Time, if you don't need to dynamically switch Adapter, you can put B A Merge back to the original weight to avoid extra matrix multiplication. This is also an important engineering selling point of LoRA compared to traditional Adapter.
Formulas and Tensor Shapes of LoRA
Let a linear layer input x shape is [B, T, d_in], original weight W0 shape is [d_out, d_in]. The full amount of fine-tuning will be updated directly W0 Middle d_out * d_in parameters; LoRA freezes W0, learn:
So the forward pass can be written as:
here r is rank, alpha/r is the scaling factor. seem d_in=d_out=4096 Further r=8, the full update requires about 16.77 million parameters, while the LoRA increment is only 8*(4096+4096)=65536 parameters. This order of magnitude difference explains why the training optimizer state is significantly smaller; but it does not mean that the activation video memory, KV cache or data loading overhead will disappear automatically.
Initialization also has reproduction meaning. The PEFT documentation indicates that the default initialization will make LoRA approximate No-OP when starting: A random initialization, B Initializes to zero. In this way, the output of the model step 0 is consistent with the frozen base, and the training will not destroy the pre-training behavior at the beginning. If both are randomized for debugging, it should be clearly marked that this is no longer a common initialization.

A guide to the official implementation
Microsoft's loralib It is the shortest path to read LoRA. Suggestion first loralib/layers.py Middle Linear: it creates lora_A、lora_B, set self.scaling = lora_alpha / r and pre-train weight.requires_grad set as False. The forward function first calculates the ordinary F.linear(x, weight), plus x @ A^T @ B^T * scaling. The training/assessment mode switching also contains the MERGE/UNMERGE logic, which is used to add low rank updates back to the original weight or revoke.
Hugging Face PEFT shows the configuration layer that will be encountered in the research reproducing experiment. LoraConfig The most likely to affect the results is r、lora_alpha、target_modules、lora_dropout and bias. If the target module name is wrong, the training parameter may be zero; if only given q_proj、v_proj plus LoRA, and QLoRA style target_modules="all-linear" Not the same experiment. The official report should print the Trainable Parameter Ratio and save the Adapter configuration.
The reading focus of the official QLoRA repository is not to copy the entire training script to the local machine, but to see how it loads the quantitative model and invoke prepare_model_for_kbit_training, and then add the LoRA module. This makes it clear one thing: QLoRA is not "training 4-bit weights", but stores/calculates the frozen base in 4-bit, and reverses the gradient to the LoRA Adapter.
QLoRA multi-provincial video memory, where is the province
There are three key increments for QLoRA: NormalFloat 4-bit (NF4), Double Quantization and Paged Optimizers. NF4 is a 4-bit data type for approximate normal distribution weights; Double Quantization continues to quantify quantization constants to reduce the average storage overhead. The instantaneous video memory peak. The paper reports that QLoRA can be used on a single 48GB GPU Fine-Tuning 65B The model, but this conclusion depends on the specific model, sequence length, batch, checkpointing, optimizer and hardware, and cannot be directly launched from the toy script.
The video memory budget should be disassembled. The full FP16 fine-tuning usually requires the base weight, gradient, optimizer state and activation; after the LoRA freezes the base, the gradient and Adam states are mainly reserved for Adapter; QLoRA Further press the freeze base weight to 4-bit. Matching script is located in code/minimal_lora_qlora_budget.py, the default is to use a 7B parameter model and 20 million LoRA parameters to make a rough ledger:
python3 code/minimal_lora_qlora_budget.py --check-only
The actual operation of the standard library inspection outputs a single layer base_weight、lora_A、lora_B、delta_W Shape, and give three rough tracked memory for Full FP16, LoRA FP16 base, and QLoRA 4-bit base. Note that it has no statistical activation, temporary kernel, allocator fragments, CUDA context, checkpoint save and tokenizer batch, so it can only be used as a draft budget before the experiment.
Minimum Experiment and Acceptance Agreement
After installing PyTorch, you can run a CPU toy training:
python3 -m pip install -r code/requirements.txt
python3 code/minimal_lora_qlora_budget.py --steps 40 --seed 7
The Toy model has only one freezing linear layer and one LoRA increment, and check whether the LoRA parameter can be optimized with the synthetic two-classification task. Acceptance should not only see if the loss drops, but also check four invariants: the freezing weight gradient is empty, and the number of LoRA parameters is equal to r*(d_in+d_out), Step 0 No-OP initialization will not change the base output, the training log records SEED, RANK, ALPHA, learning rate and BATCH SIZE.
If it is extended to the real model, it is recommended to divide the evaluation protocol into three layers. The first layer is the configuration recheck: model name, revision, tokenizer, dataset version, target modules, rank, alpha, dropout, and quantitative configuration are all written to the log. The second layer is that the resources can be reviewed: peak video memory, time per step, valid batch, sequence length, whether Gradient Checkpointing is enabled. The third layer is the task indicator: Validation Set Loss, Accuracy, MT-Bench or Vicuna. The dynamic list and closed-source evaluator versions must be marked with "waiting for manual verification".
How to report reproducing experimental boundaries
LoRA/QLoRA reproducing reports in particular need to separate "configuration success" and "methods are effective". The successful configuration only shows that the Adapter has been inserted, the parameters are in line with the expectations, and the training cycle can run through; the method is effective also needs to compare the indicators under fixed data segmentation, fixed evaluation script and reasonable baseline. The reason why many failure records cannot be reviewed later is because only the final checkpoint is saved, but the adapter configuration, quantization configuration and training log are not saved.
It is recommended to preserve at least five types of materials in each experiment. The first is config.json Or command line parameters, which should include rank, alpha, dropout, target modules, whether to train BIAS, and whether to merge Adapter. The second is environmental information, including PyTorch, Transformers, PEFT, BitSandbytes, CUDA, GPU models and driver versions. The third is the resource log, at least record peak video memory, time-consuming time, sequence length, batch size and gradient accumulation. The fourth is data records, including dataset version, filtering rules, template, maximum length and random seeds. The fifth is the evaluation output, it is best to keep the sample-by-sample prediction or score the details, rather than just an average score of one line.
Take care when citing the reported results. The LoRA paper compares parameter efficiency and model quality on RoBERTa, DeBERTa, GPT-2, GPT-3, and other models. The QLoRA paper reports memory requirements for 4-bit fine-tuning of large models and results for dialogue models. Attribute those findings to the paper or official repository. The local checks establish only that the script can verify LoRA parameter shapes, calculate parameter ratios, and estimate memory use without PyTorch. The toy PyTorch run and actual QLoRA training still need verification with the required dependencies and hardware.
Another often overlooked boundary is randomness. There are very few LoRA parameters, which does not mean that the experimental variance must be small. Small datasets, short training steps, different prompt templates, and different tokenizer truncation strategies may make the final indicator fluctuate significantly. Reliable reports should run at least multiple SEEDs, or explicitly label single SEED results as exploratory results. If the budget is not enough to SEED, you should also give priority to keeping the loss curve, the Validation Set output and the failure sample, rather than just keep the best screenshot once.
Failure checklist
First, the training parameters are zero. Usually target_modules The real layer name is not matched, or the trainable parameter is not printed after the Adapter injection. The projection layers of different architectures are named differently, and LLaMA's q_proj Put it to all models as it is.
Second, there is no significant decline in memory. The common reason is that embedding, LM head, or BIAS are still being fully updated; the sequence is too long to cause activation as the main overhead; or the Gradient Checkpointing is not enabled. LoRA mainly reduces trainable weights, gradients and optimizer states, and does not eliminate forward activation for free.
Third, the QLoRA quantitative configuration is inconsistent. Typical configurations in Hugging Face documentation include load_in_4bit=True、bnb_4bit_quant_type="nf4"、bnb_4bit_use_double_quant=True and suitable compute dtype. The memory and speed may vary if the library version, hardware backend, or dtype is different.
Fourth, continue training after the Adapter merge. MERGE is suitable for reasoning or export; if the training phase is repeated merge/unmerge, make sure that the low rank update is not repeated to the weight. Official implementation of maintenance merged state, to avoid such errors.
Fifth, the reporting method is confusing. The 65b single-card conclusion of the paper report, the Guanaco results of the official repository, and the parameter check of the local Toy Script are three kinds of evidence. Scientific research blogs must be written separately: what the paper claims, what is actually verified this time, and what the author deduces according to this.
Sixth, ignore the merge check. Before and after adapter merge, the logits difference should be compared on the same input; if the difference is common, the dtype, scaling factor, whether repeated merge and weight transfer direction should be checked first. This inspection is very cheap, but it can detect serious problems in the export stage in advance.
The finer the reproducing log, the lower the subsequent errors.
Questions for further study
- Under the same training budget, is the Validation Set gain and the memory increment linear when LoRA Rank goes from 4, 8, 16 to 64 ?
q_proj/v_proj, attention full projection, MLP full linear layer andall-linear, is the difference from capacity or the location of the target module?- How do NF4, FP4, and INT8 compare the confusion degradation and training stability on the same model and the same data?
- Is the maximum error of Logit acceptable under different dtypes before and after LoRA Adapter merge?
- In the small data instruction fine-tuning, is LoRA's overfitting performance lighter than the full amount of fine-tuning, or is it just that there are few training parameters that lead to slow convergence?
These problems are more suitable as research entrances than "can you save video memory", because they put methods, budgets, indicators and failure patterns in the same reproducible framework.
Summary
The minimum reproduction of LoRA can be condensed into one sentence: Freeze W0, training A and B, with B A Approximate the weight update required for downstream tasks. QLoRA quantizes the frozen base to 4-bit on this basis, and continues to train only LoRA Adapter. After understanding these two points, the video memory budget is no longer a metaphysics: base weight, trainable parameters, optimizer status, and activation peak must be estimated separately.
This article completes the source code reading, formula alignment, budget script and official drawing. PyTorch has not been installed on this machine, so the TOY training and real model QLoRA are still labeled for manual verification. The next most worthwhile step is to compare the video memory, speed and verification metrics of Full Fine-Tuning, LoRA and QLoRA after the fixed model Revision and the dataset is split, instead of just reporting a beautiful Adapter parameter ratio.
References
Search date: 2026-08-26. The following links are preferred to use paper pages, official repositories and official documents; PEFT, Transformers and BitSandbytes documents will change with the version, and the official reproduction experiment should record the local package version, GPU model and model. revision.
- Edward J. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, arXiv 2021 / ICLR 2022.
- Microsoft, LoRA official repository,
loralibPyTorch implementation. - Microsoft,
loralib/layers.py, LoRA Linear, Merge/Unmerge and freeze weight logic. - Tim Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, arXiv 2023 / NeurIPS 2023.
- Artidoro Pagnoni et al., QLoRA official repository, training scripts, configuration and Guanaco instructions.
- Hugging Face PEFT, LoRA documentation,
LoraConfig, initialize and target modules. - Hugging Face PEFT, Quantization guide, LoRA and 4-bit/8-bit quantization process.
- Hugging Face Transformers, bitsandbytes quantization documentation,
BitsAndBytesConfig, NF4 and Nested Quantization. - Hugging Face bitsandbytes, Linear4bit / LinearNF4 reference, 4-bit linear layer and NF4 API description.