In the previous few articles, we have moved forward this main line to this:

  • Why Transformer can become a large model infrastructure

  • What does a model learn during pretraining?

  • Why is it necessary to align training such as SFT, RLHF, and DPO?

  • How the long context is made

  • How to adjust the inference parameters such as temperature, top-k, top-p, repeat penalty

Next, we will enter a very "engineering" topic:

When you already have a base model in your hand, how should you transform it into a model that suits your task?

When many teams do large-model training for the first time, they are most likely to fall into two extremes:

  • Either think "Since you want to adjust, just refer to all fine-tuning"

  • Either think "LoRA must be the most cost-effective, all tasks should be LoRA"

But the real situation is not a choice, but a balance of resources, goals, risks and returns.

In this article, I want to make the three most common routes clear:

  • What kind of scenarios is suitable for Full Fine-tuning (Full Fine-tuning)

  • Why can LoRA use very few parameters to change the obvious effect?

  • Why does QLoRA become the mainstream solution in the scenario of single-machine resource restriction

And, more critical engineering issues:

  • What are the parameters of rank, alpha, dropout, and target modules

  • 4bit / 8bit, gradient checkpointing, batch size, learning rate

  • How to choose single card, consumer grade graphics card, multi-card server

If you are preparing to do industry Fine-Tuning, private data adaptation, vertical task alignment or multimodal follow-up training, this article will be more useful than "memorize a few nouns".


1. First, let’s talk about the conclusion: The Fine-tuning plan is not selected according to the "advanced level", but according to the goal.

Correct a misunderstanding first.

It’s tempting to see LoRA, QLoRA, and full fine-tuning as successive upgrades, as if having enough resources automatically made training every parameter the right choice.

This is actually not right.

A more accurate understanding is:

They are model adaptation problems under different constraints.

You can remember this frame of judgment first:

  • If you mainly want the model to learn a certain output format, style, and domain question and answer habits, first look at LoRA / QLoRA

  • If you want to significantly change the boundaries of the model's ability, such as large-scale continuous pre-training, deep domain migration, and the structural offset of the model, consider the full reference of fine-tuning

  • If your bottleneck is first of all video memory, not training time, QLoRA is usually the most realistic starting point

  • If your data volume is very small, it is easier to overfitting and catastrophic forgetting

So the first question in the project is not "which method is the strongest", but:

Is this training doing ability injection, style alignment, or ability reshaping?


2. Full reference fine-tuning: the upper limit of effect is high, but the resources and risks are also the highest

The meaning of full reference to fine-tuning is very direct:

The trainable parameters in the model are all participating in the update.

For a 7B model, full fine-tuning means backpropagating through all roughly 7 billion parameters, storing their gradients, and maintaining optimizer state for them.

This leads directly to three results:

1) Advantages: the most complete expression ability

Because all layers can be updated, the model can adapt to the new distribution more thoroughly.

The scenarios it fits typically include:

  • Continued pretraining

  • Mass migration, such as general corpus cut to legal, financial, biomedicine

  • It is necessary to significantly change the internal representation of the model, not just the output behavior

  • A team with sufficient data, computing power and experimental budget

2) Disadvantages: video memory and training costs are very high

It is not only the model parameters themselves that really eat resources during training, but also:

  • Gradient

  • Optimizer State (especially Adam class optimizer)

  • activation

A very rough but engineering useful experience is:

The video memory usage of the full participation training is often several times that of the "model weight video memory".

Therefore, it is also a 7B model, and reasoning can run, which does not mean that training can also run.

3) Risk: It is easier to forget the original ability

If the data distribution is narrow, the learning rate is unstable, or the epoch is too large, all references are easy to appear:

  • The original general ability has declined

  • The output style narrows

  • Performance fallback for some basic tasks

This is why many business fine-tuning scenarios are not in a hurry to fully refer to them.

Because what you really want is often not "changing the model", but "adding a layer of controllable adaptation on the premise of not destroying the original model as much as possible".


3. LoRA: Why can we call the model out with a small number of parameters?

The core idea of LoRA is to write the original huge weight update into a low rank decomposition.

Intuitively, it can be understood as:

Instead of directly modifying the entire large matrix, learn a smaller, structurally limited increment.

The benefits of doing this are very clear:

  • Original model weights can be frozen

  • The amount of trainable parameters has dropped significantly

  • Significantly reduced memory pressure

  • You can save multiple sets of Adapters for different tasks, and the switching cost is very low

This is also the reason why LoRA can quickly become the mainstream method.

For many instruction-tuning, domain-specific question-answering, and output-format tasks, a few targeted updates to key projection layers suffice. You don’t need to rebuild the model’s internal knowledge from scratch.

LoRA fits the needs just right.

Where is the most common plug of LoRA?

In Transformer, common target modules include:

  • q_proj

  • k_proj

  • v_proj

  • o_proj

  • gate_proj

  • up_proj

  • down_proj

In many practices, you can already get good results by doing LoRA to Q/V or Q/K/V/O in attention.

If the task is more complex, or if you want the model to adapt to the new field more deeply, the FFN-related layer will be included together.

Experience:

  • Only hit the attention layer, the training is more economical, suitable as a starting point

  • Attention + FFN is stronger, but the amount of parameters and training costs will increase


4. QLoRA: Why is it especially suitable for the scene of "the graphics card is not luxurious but want to do fine-tuning" seriously

QLoRA can be seen as:

Quantified base model + LoRA adapter training.

Its key point is not to "quantize LoRA too", but:

  • The base model is loaded in 4bit to reduce memory usage

  • What is really updated during training is still LoRA Adapter

  • Through the paging optimizer, quantization weight return and other skills, the single machine can also train a larger model

What does this mean?

It means that only high-end multi-card machines can do fine-tuning, but now many consumer-grade GPUs can also start trying.

For example, in practical engineering, QLoRA is often used:

  • Single Card 24GB Graphics Fine-Tuning 7B / 13B Weight Model

  • Small teams can quickly verify whether industry data is effective

  • Iteration Prompt + Data + Adapter combination in low budget scenarios

But QLoRA is not without a price.

Its main cost is usually reflected in:

  • The training throughput is slower than pure FP16 LoRA

  • Stability is more dependent on implementation details under some extreme configurations

  • If the data is large and there are many training rounds, the final upper limit may not be better than a higher resource plan

So the essence of QLoRA is not the "strongest solution", but:

Under cost constraints, a cost-effective solution is very high.


5. What are the most critical parameters, what is the control?

When many people start training, the first confusion is these parameters. The following is a direct description of the meaning of the project.

1)rank(r)

rank determines the capacity of the LoRA increment matrix.

It can be understood as "how much space you allow Adapter to have".

Common experience:

  • r = 4 / 8: more resource-efficient, suitable for lightweight style alignment, format learning

  • R = 16 / 32: more common, suitable for most domain adaptation tasks

  • r = 64 and above: more expressive ability, but the risk of video memory, training time, and overfitting will increase

It's not that the bigger the RANK, the better.

If the amount of data is limited, too high rank tends to just make the model remember the training set faster, rather than better generalization.

2)alpha

Alpha can be understood as the scaling factor of the LoRA update amount.

The common practice is to make alpha and rank on the same scale, or slightly larger than the rank, such as:

  • r = 8, alpha = 16

  • r = 16, alpha = 32

It affects the strength of the Adapter update to the base.

If the training is very unstable, or there is obvious overfitting, you can look back together with the learning rate, instead of just staring at the loss.

3)LoRA dropout

Its function is similar to ordinary dropout, which is to prevent Adapter Overfitting .

Common intervals:

  • 0: Common when the amount of data is large and the task is stable

  • 0.05

  • 0.1

If your sample size is small and the style is very concentrated, appropriate dropout is often more effective than blindly dropping the learning rate.

4)target modules

This is one of the most underestimated parameters that affect the training effect.

Which layers you add LoRA to determine essentially which capabilities the model is allowed to adapt to.

Experience suggests:

  • Start with the Attention projection layer

  • If it is found that model learning cannot be expressed in complex fields, expand to the MLP layer

  • Don't come up to "full-level random fight", first do a controllable experiment

5)learning rate

LoRA / QLoRA can often use a higher learning rate than all-parameters fine-tuning, but this is not an absolute rule.

Common starting points:

  • LoRA / QLoRA: 1E-4 to 2E-4 common

  • All-in-Tuning: 1e-5 to 5e-5 is more common

If the task is conservative, the data quality is average, and the sample size is not large, I would recommend starting with a smaller learning rate.

Because in the large model fine-tuning, the most expensive thing is not "more training for one more day", but the direction is found in the wrong direction after training.

6)batch size / gradient accumulation

What really matters is not the Micro Batch written in your configuration, but:

effective batch size.

That is:

effective batch=per device batch×gradient accumulation×GPU 数\text{effective batch} = \text{per device batch} \times \text{gradient accumulation} \times \text{GPU 数}

When there is not enough video memory, the most common practice is not hard descending sequence length, but:

  • Reduce Micro Batch first

  • Then use the gradient accumulation to make up the valid batch back

This is one of the most practical tuning methods in stand-alone training.


6. How to choose resources: what are the suitable solutions for single card, multi-card and server

Here is a very practical engineering judgment.

1) Consumer-grade single card (such as about 24GB)

The more realistic starting point is usually:

  • 7b magnitude model

  • QLoRA or LoRA

  • Turn on Gradient Checkpointing

  • Reasonable control Sequence length

This type of configuration is best for:

  • Industry Q&A adaptation

  • The internal knowledge-style migration of enterprises

  • Structured output optimization

2) Stand-alone multi-card / medium training resources

At this point you can start thinking:

  • LoRA with a higher rank

  • longer context training

  • Adapter Fine-Tuning of 13b or even larger models

If the target is business adaptation, the LoRA system is usually still more cost-effective.

3) High budget server cluster

Only when you meet the following things at the same time, all references to fine-tuning are more worthy of serious progress:

  • The amount of data is large enough

  • Clearly know that it is not just the output style to change

  • There is a complete evaluation system to monitor forgetting and degradation

  • Can bear multiple rounds of training and rollback costs

Otherwise, many so-called "more advanced" decisions will only make mistakes more expensive in the end.


7. Deployment perspective: the training method will directly affect the online method

This matter is often overlooked.

Many teams only look at the loss during training, not how to go online at the end, but only when they reach the deployment stage are they found that the plan is not smooth.

Deployment Benefits of LoRA / QLoRA

  • Adapter files are small for easy version management

  • The same base model can hang multiple sets of tasks Adapter

  • Suitable for grayscale, A/B and fast rollback

  • Friendly to multi-tenant or multi-business line scenarios

Deployment cost for full reference to fine-tuning

  • Each version is a full copy of the model

  • Higher storage and distribution costs

  • Rollback is heavier

  • Model governance and version auditing are more complex

If you are doing production services, not academic Benchmark, Deployment costs themselves are part of the training scheme choice.


8. The most common pit: it is not that the training cannot run, but the goal is not clearly defined

Finally, a few of the most common questions.

1) The amount of data is very small, but blindly pursue a big Rank

The result is usually the training set works well, and the Validation Set and the real online scene are a mess.

2) Just look at the loss, not the task indicators

In many SFT/LoRA tasks, the loss of loss does not mean that the structured output is more stable, and it does not mean that the fact is more consistent.

Be sure to check downstream indicators, such as:

  • Format Accuracy

  • fact consistency

  • command compliance

  • Whether the behavior of abstention is abnormal

3) Misjudging "insufficient domain knowledge" as "fine-tuning is not enough"

Some problems are actually more suitable for RAG supplement knowledge, rather than forcibly put new information into the model parameters by force-tuning.

If the knowledge update is frequent and the version changes rapidly, the search enhancement is preferred, and do not use fine-tuning as a database.

4) Regardless of the context length during training, long input is required during deployment

The distribution of training and deployment is too large, and the final effect is often unstable.

If the long document, long question and answer, and multiple rounds of dialogue will be processed when it is online, the corresponding distribution should be covered as much as possible during the training stage.


9. How to choose: give you a decision table that can be directly implemented

If you just want to get a practical conclusion first, you can directly judge this:

Prefer the case of LoRA

  • The goal is field adaptation, format learning, style adjustment

  • I hope that the training is fast, the deployment is light, and the version switching is convenient

  • Small data scale

  • Do not want to obviously destroy the general abilities of the base model

Prefer the case of QLoRA

  • Video memory tension

  • Only single card or limited GPU

  • Need to verify task feasibility under low budget

  • The training speed can be slightly slower, but the threshold must be lowered

Consider the case of full reference to fine-tuning

  • There is a large number of high-quality field data

  • The goal is deep capability migration, not just output alignment

  • Sufficient training resources and evaluation system

  • Can withstand higher deployment and rollback costs

If I had a universal advice, it would be:

90% of the business fine-tuning projects should first strengthen the baseline from LoRA / QLoRA, rather than do full fine-tuning.

Because what really determines the success or failure of a project is usually not "you use the heaviest training method", but:

  • Is the data clean

  • Is the task definition accurate?

  • Evaluate whether the loop is closed

  • Is it controllable online?


10. Conclusion: fine-tuning is not "more ruthless" than who, but "modified more accurately" than who

The most exciting thing in large-model training is the amount of parameters, the number of graphics cards and the length of training.

But what can really make a project is often not these surface indicators, but whether you figure it out:

  • Which abilities should be obtained by fine-tuning

  • What knowledge should be provided by RAG

  • What problems are actually the inference parameters and system design problems

LoRA, QLoRA, and full reference are not a struggle for faith.

They are just engineering tools under three different resource constraints.

Choosing the right tool is much more important than a heaper tool.

Next, I will continue to write down:

Large language model Deployment actual combat: how to choose FP16, INT8, 4bit quantization? Throughput, precision and real trade-off of video memory.

This article will include another practical problem after training:

The model can be transferred out, which does not mean that it can run cheap, stable and large-scale.

If you are preparing to make your own model adaptation, I hope this article can help you take the detour of "insufficient video memory, unstable effect, and troublesome online".