In the previous articles, we have pushed this main line to the fine-tuning stage:

  • Why Transformer can become a large model infrastructure
  • What does a model learn during pretraining?
  • How to string alignment training such as SFT, RLHF, DPO
  • How the long context is made
  • How to adjust the inference parameters such as temperature, top-k, top-p, repeat penalty
  • What scenarios are suitable for LoRA, QLoRA, and all parameters fine-tuning

Next, it's time to enter a deployment problem that really decides whether you can run.

Because when many teams move large models from papers, demos, and notebooks to the real environment for the first time, the first collision is not Prompt, nor is it an algorithm, but a very real problem:

The memory is not enough.

You'll find that many things can't get around it:

  • Can the 7B model run on a single card?
  • How much video memory do the 13b, 32b and 70b models need?
  • Why can some models FP16 not run, but can run with 4bit?
  • INT8, 4bit, GPTQ, AWQ, GGUF, bitsandbytes how to choose?
  • Why does it sometimes save video memory after quantization, but not necessarily faster?
  • In the real service, should we give priority to ensuring the throughput and precision, or can it be launched first?

In this article, I would like to explain the most critical line in the deployment of the large model:

Quantization is not as simple as "compressing the model", but engineering trade-offs between video memory, precision, delay, throughput, and compatibility.

We focus on 7 things:

  1. Why Quantization is the Core Leverage of Large Model Deployment
  2. What do FP16, BF16, INT8, and 4bit mean respectively?
  3. What is the difference between GPTQ, AWQ, GGUF, Bitsandbytes
  4. Why "Lower Bit" doesn't necessarily mean "faster"
  5. Why do KV cache, batch, and context lengths explode the video memory again?
  6. How to choose a quantitative solution in the three scenarios of stand-alone, local and API service scenarios
  7. How to do a real evaluation of throughput, delay, and precision, instead of just looking at the parameter table

If you are about to put the model into:

  • local development machine
  • Consumer Grade GPU Server
  • Reasoning Service Cluster
  • Frameworks like Ollama/vLLM/SGLang/LLAMA.cpp

This article will be more useful than "memorizing a few quantitative terms".


First, let's talk about the conclusion: the first solution to quantization is "whether it can be deployed", and the second is "how beautiful it runs"

When many people first come into contact with quantization, they naturally understand it as:

  • The model is smaller
  • Save a little bit
  • Faster

This is of course true, but not accurate enough.

A more practical understanding should be:

Quantization is the first engineering lever that pulls back a model that cannot be deployed back to the deployable area.

Because in real environments, deployment constraints are usually not "theoretical optimal", but the following:

  • You only have 24GB, 48GB or 80GB graphics card in your hand
  • You also have to leave KV cache, batch, concurrency and framework overhead
  • You can't just let it run once, but to handle requests stably
  • Not only do you want to produce results, but also control TTFT, tokens/s, throughput and cost

So the first question in the project is usually not:

Which precision is the most advanced?

but:

Under my hardware and target SLA, what format is the most suitable for the model.

If a 14B model won’t fit in memory in FP16, its accuracy at that precision is beside the point for that deployment.


2. Three distinct concepts: weights, activations and the KV cache

When many people just learn to deploy, they will understand the "model video memory" as a static number.

This is easy to be misjudged.

Because AT Inference Time memory is usually divided into at least three pieces:

1) Model Weights (Weights)

That is the parameter itself.

This is the most direct part of quantization.

For example:

  • FP16: about 2 bytes per parameter
  • INT8: about 1 byte per parameter
  • 4bit: about 0.5 bytes per parameter

Rough estimate:

  • 7b model, FP16 weights about 14GB
  • 7b model, INT8 weight about 7GB
  • 7B model, 4bit weights about 3.5GB

But this is just a very rough theoretical value, and the truth should be added:

  • Scale/zero-point and equal quantization metadata
  • Framework overhead
  • memory fragmentation
  • tensor parallel / runtime buffer

2) Activations

AT Inference Time activation overhead is usually much smaller than training, but it cannot be completely ignored in large batch or some framework implementations.

3)KV Cache

This is where many people are really stunned after they go online.

Because generative reasoning is not just loading the model in, the model also needs to save the attention history for each request, that is, KV Cache.

The occupation of KV cache is strongly related to these factors:

  • layers
  • hidden size
  • attention heads
  • context length
  • batch size / concurrent number
  • Precision Format

So there is often a situation:

The model body can be put down, but once the BATCH is slightly larger, the context is elongated, and the concurrent comes up, the video memory is still exploded.

This is why the deployment discussion should not only focus on "how big the model is", but also " how much the runtime".


3. What are FP16, BF16, INT8, 4bit?

Make these common formats first.

1)FP16

FP16 is a semi-precision floating-point, the most classic format in deployment.

Its features are:

  • Precision is higher than low bit quantization
  • Framework compatibility is generally better
  • Many GPUs have mature acceleration of FP16 inference
  • Video memory occupancy is about parameter amount × 2 bytes

It fits:

  • you have enough video memory
  • You are more conservative and stable
  • You don't want to introduce too many quantitative errors
  • You're going to be a benchmark or a high-fidelity baseline

2)BF16

The BF16 is essentially 16 bits, but the index bit is more, and the numerical stability is usually better than the FP16.

BF16 is common in modern training and reasoning frameworks, especially on GPUs that support good.

From the deployment perspective, its core commonality with FP16 is:

All belong to the "high-precision, relatively large video memory overhead" solution.

3)INT8

INT8 is an 8-bit integer representation.

Its intuition is simple:

  • Parameters occupancy is about halved
  • Often saves video memory than 16bit
  • Accuracy loss is usually milder than 4bit

It is often a good middle ground:

  • Want to save resources
  • I don't want to endure too aggressive reduction in precision
  • Hope to retain strong compatibility and stability

4)4bit

4bit is a very popular one in open source deployment now.

Because it can often pull the model that "can't run at all" into the running range of the consumer-grade graphics card.

Its value is very realistic:

  • The larger model can be installed in the same video memory
  • The threshold for local deployment has dropped sharply
  • Especially useful for models like 7B, 14B, 32B

But remember:

The income of 4bit is first of all "installation", not automatically guaranteed "faster, more stable and more accurate".


4. Why is lower bit not necessarily faster?

This is the most misunderstood place for many people.

Intuitively you will feel that:

  • The model is smaller
  • Less data handling is less
  • So it must be faster

But the real situation is not so simple.

Because the inference speed depends not only on the weight size, but also on:

  • Quantization and dequantization overhead
  • Whether Kernel is optimized to mature
  • Does the GPU support this precision format well?
  • Whether the framework has made efficient implementation of this format
  • The current bottleneck is computing power, bandwidth or scheduling overhead

So there are three common situations:

1) Low Bit and save memory and faster at the same time

This usually happens at:

  • The model was originally limited to the bandwidth of the video memory
  • Quantify Kernel well done
  • Hardware support is better
  • The request scale is more suitable for the current framework

2) Low Bit saves video memory, but the speed improvement is limited

This is actually very common.

Because you just solved the "can run" problem, but the throughput still suffers:

  • KV Cache
  • Dispatching expenses
  • Decode Stage Seriality
  • Framework Efficiency

restrictions.

3) Low bit saves video memory, but may not be faster

On some CPUs, in local inference, or with less mature implementations, excessive dequantization and format conversion can eat up the savings.

So don't understand the simplicity of quantization as "the lower the bit is, the faster it is".

A more accurate understanding is:

Lower bit widths give you more deployment options. The speedup depends on the hardware and inference stack.


5. What is the difference between GPTQ, AWQ, GGUF, BitSandbytes, etc.?

Instead of pursuing the details of each paper, you need to know the difference you most need to know from an engineering point of view.

1)bitsandbytes

This is the first stop for many people who are exposed to 8bit/4bit quantization.

Its features are:

  • Combine with Hugging Face ecology
  • Easy to load
  • Suitable for rapid experiment and development verification
  • Commonly used for QLoRA, low-threshold local deployment

Suitable for scenarios:

  • Do you want to quickly try a model to run?
  • You mainly work in Python/Transformers ecological work
  • You pay more attention to development convenience, rather than extreme inference performance

2)GPTQ

GPTQ is a typical post-training quantization (PTQ) route.

Its goal is:

In the case of retraining the model, measure the weight to a lower bit, and at the same time minimize the loss of precision.

Suitable for scenarios:

  • You already have a trained model
  • Want to turn it into a low-bit version suitable for reasoning deployment
  • Pay more attention to the deployment products after offline quantization

3)AWQ

AWQ is also one of the very mainstream post-training quantization routes at present.

It emphasizes the more targeted protection of important weights, so it has good precision to maintain performance in many models and tasks.

Many people like it in engineering because it is relatively balanced between "precision-speed-deployment".

Suitable for scenarios:

  • You have to go online GPU Reasoning Service
  • You want to drop as little as possible after 4bit quantization
  • You want to cooperate with more mature service stacks such as vLLM/SGLang/TensorRT-LLM

4)GGUF

GGUF is more of a very common model distribution format under the LLAMA.cpp system.

Its value is not just quantization, but:

  • Convenient local reasoning
  • CPU / Apple Silicon / Small device support is good
  • Rich in various quantities
  • Very friendly to desktop, self-hosted, lightweight local use

Suitable for scenarios:

  • Mac local running model
  • CPU inference
  • Do not pursue large-scale service, but pursue convenient deployment

So you can understand this first:

  • BitsAndBytes: Development Authentication Friendly
  • GPTQ: Classic Offline Training Quantization Route
  • AWQ: Deploy a very common cost-effective 4bit solution in practice
  • GGUF: Strong localization, cross hardware, lightweight deployment

6. Quantitative selection should not only look at the weight, but also count the kv Cache together

This is a common source of deployment errors.

When many people estimate the memory, they only count:

Model parameter quantity × number of bits

Then I found that the theory is enough, and the actual run is still OOM.

Why?

Because in real reasoning, especially in long contexts and high concurrency scenarios, KV cache often becomes the second largest video memory consumption source.

a practical judgment

If your scenario is:

  • Long Document Questions and Answers
  • RAG answers a lot of context when answering
  • High Concurrent API Services
  • Multiple rounds of sessions keep context for a long time

Then you have to give priority to not only the model quantization gear, but also include:

  • max model len
  • max num seqs / batch size
  • prefill length
  • Decode stage concurrency
  • KV cache precision and paging management

That is:

Quantization solves the static model occupation, and the KV cache determines the dynamic service capacity.

If you ignore the second thing, the deployment is easy to roll over after going online.


7. How to choose three types of scenarios, stand-alone, local and service?

I give you a selection framework that is closer to real engineering.

1) Local personal use / development verification

The target is usually:

  • Run first
  • Low cost
  • Do not pursue limit throughput

Recommended ideas:

  • Mac/CPU/ Lightweight Devices: Prioritize GGUF
  • Single-card experiment: you can try bitsandbytes 4bit / 8bit
  • If it's just Prompt debugging and functional verification, don't pursue the most complex service stack

The most important thing at this time is:

Let the model enter the Availability Zone, rather than full the architecture at one time.

2) Team intranet service / small-scale API

The target is usually:

  • Can be called by multiple people
  • have some concurrency
  • Stability is more important than limit parameters

Recommended ideas:

  • 7B / 14B model: INT8 or 4bit are worth evaluating
  • If you are sensitive to precision, first use FP16/BF16 as baseline, and then compare AWQ/GPTQ
  • Use vLLM/SGLang for a more service-friendly reasoning framework
  • Focus on monitoring TTFT, tokens/s, OOM, context length and concurrent degradation

3) Production-grade high concurrency service

The target is usually:

  • as high as possible
  • Predictive delay
  • Stable SLA
  • cost controllable

At this time, the quantitative choice should not only look at "can run", but look at:

  • Whether a specific framework supports this quantization format is mature
  • Continuous batching can eat full
  • How Tensor Parallel / Pipeline Parallel Collaborate
  • Does the drop in accuracy affect real business indicators?
  • Is it convenient to fail recovery, heat loading, and grayscale switching?

In other words, The online service looks at the entire reasoning chain, not a single quantization label.


8. How to evaluate whether "quantization is worth it"? Don't just look at subjective feelings

This part is particularly critical.

When many teams measure, just ask one question:

  • Doesn't seem to be too different?

It's not enough.

You should evaluate at least from four dimensions.

1) Video memory usage

Record:

  • Memory after the model is loaded
  • first token front video memory
  • long context video memory
  • Peak memory at high concurrency

2) Delay

Focus on:

  • TTFT(Time To First Token)
  • Total delay of single request
  • Changes under different Input Length

3) swallow

Focus on:

  • tokens/s
  • BATCH throughput changes after growth
  • The degradation curve after the concurrency rises

4) Accuracy / business effect

Don't just read one sentence "It feels okay to answer".

should at least be prepared:

  • Small sample regression test for generic benchmark
  • Your Real Business Sample Collection
  • Hallucination rate, format compliance, recall hit, code executability and other task indicators

Because many quantization solutions can't see the difference in chat, but in:

  • Mathematical Reasoning
  • structured extraction
  • Strict format output
  • OCR / Documentation Understanding
  • Multiple tool calls

These tasks are more likely to expose errors.


9. Several practical parameters and engineering suggestions

Finally, I will give you a set of more landing experience judgments.

1) First use a high-precision format to establish a baseline

If the hardware allows, first run the FP16/BF16 baseline.

In this way, when you evaluate INT8 and 4bit later, you will know where the drop and the benefits come from.

2) Don't just stare at the weight of the model, remember to leave a budget for the KV cache

Especially long context and multi-concurrency service.

Many OOMs do not happen at the moment of "loading the model", but after "requests are more".

3) The more aggressive the quantization is, the more it is necessary to do real business return

Don't just take the universal question and answer test.

If your business is:

  • RAG
  • document extraction
  • code generation
  • understanding of the table
  • Multimodal Q&A

Then use these real tasks to return.

4) If the goal is service, don't just look at the model file format, look at the support of the entire reasoning framework

The same 4bit model, on different frameworks:

  • Swallow and spit
  • Stability
  • Concurrent support
  • contextual ability

There may be a very big difference.

5) The "optimal quantitative solution" is usually not fixed, but is bound to the model, hardware, and task.

There is no format that wins in all cases.

Work to avoid asking:

"Is 4bit must be the best?"

More should be asked:

"Which of my model, hardware, and task combination is the most suitable?"


Finally, a summary

In today's article, you can write down the core into the following sentences.

1) Quantization is the first solution to the deployment feasibility

In many cases, it is not "run better", but first let the model enter the interval of "can go online and carry requests".

2) FP16 / BF16 are more conservative, INT8 is the middle zone, 4bit is the best at compressing video memory

But the lower the bit, the faster and more stable the automatic.

3) GPTQ, AWQ, GGUF, and bitsandbytes serve different inference stacks and use cases, so they aren’t interchangeable

Development validation, GPU service, local reasoning, each optimal solution is often different.

4) Deployment evaluation must be viewed at the same time: video memory, delay, throughput, business effect

It is not enough to just see if the model can be loaded.

5) Whether a deployment succeeds also depends on the KV cache, context length, batch size, and how the framework is implemented, alongside weight quantization

These things often determine the real capacity more than the "parameter bit number".


If you want, I can directly continue to write the next article:

"Large Language Model Deployment Practice: From Ollama, vLLM to SGLang, how do local services work? 》

This quantitative content will be followed by:

  • How to choose the model format
  • How to start a stand-alone service
  • How to match Max Model Len, Tensor Parallel, Batching
  • How to implement the three routes of local development, intranet deployment and formal service respectively