In the previous article, we disassembled the underlying link of "how the image is connected to the language model".

Now you can take a step forward and enter the real concern of many people:

The same is VLM, why are the actual experience of LLaVA, Qwen-VL, miniicpm-v, and InternVL, the actual experience will be so much worse?

Some models are more like "adding picture-viewing capabilities to LLM", which is suitable for general graphic questions and answers.

Some models are obviously better at:

  • Documentation
  • OCR text reading
  • Comparison of multiple pictures
  • High resolution image input
  • Mobile or Edge Deployment

There are also some models that are strong on Benchmark, but are exposed as soon as they go online:

  • The pressure of the video memory is too high
  • Input resolution limit is dead
  • Too many pictures and texts
  • OCR stability is average
  • Highly latency in multi-picture scenarios

It's not just "variable sizes" behind this.

More importantly, they make different choices in four places:

  1. How to choose the visual front end: CLIP, SigLIP, InternViT or self-developed vision backbone
  2. How to get visual information to LLM: MLP projector, cross-attention, or more complex connection structure
  3. How to do the input strategy: fixed resolution, dynamic resolution, cut, tile, multi-map organization
  4. Where is the training objective and data distribution biased?: Universal dialogue, partial OCR document, high resolution understanding, or lighter deployment

What I want to do in this article is not to list a bunch of model names, but to make it clear from an engineering perspective:

What is the difference between the design of the mainstream VLM route, and how these differences finally affect the accuracy, throughput, delay, video memory and landing availability.

If you are choosing a type, or preparing to build your own graphic system, this article is more useful than just looking at the leaderboard.


First, give the conclusion: the difference between mainstream VLM, the core is not "will you look at the picture", but "how to see, how much, how to hand it to LLM after reading"

Many introductory discussions are easy to think of the VLM model as the same thing.

As if the difference is just:

  • Die sizes are different
  • Different parameters
  • Benchmark scores are different

However, the more critical issues are actually these three:

  • What kind of visual representation is the image coded first
  • In what form do these visual representations enter the language model
  • Whether the model is trained to handle complex tasks such as high resolution, multi-graphics, documents and OCR

These three things will directly determine the final experience.

For example, two model parameters are close, but if one:

  • Only supports medium-resolution single map
  • Access with Simple MLP Projector
  • The training data is mainly caption and general-purpose QA

Another:

  • Support dynamic high-resolution input
  • Dedicated data enhancement for documentation and OCR
  • Input organization and visual token compression is more aggressive

Their performance in real business may not be a species at all.

So look at the mainstream VLM, don't just ask:

"Which model is stronger?"

And to ask:

"What kind of visual tasks does it optimize and what cost to it?"


2. Set up an analysis framework first: compare VLM, I recommend looking at these five layers

If you are going to continue comparing the multimodal model in the future, I recommend a fixed look at five layers.

1) Vision encoder layer

Concerns include:

  • What kind of vision is used backbone
  • How big is the patch size
  • How high is the input resolution of the native supported input
  • Whether it is biased towards general semantics, or biased to document/OCR details

This is the model "the base of the picture".

2) Modal bridge layer

Concerns include:

  • Is it a simple linear mapping or a multi-layer MLP
  • Have you used cross-attention?
  • Is there a query-based compression structure
  • Whether the visual token will be resampled or compressed

This determines whether the visual information can be transmitted efficiently and stably to the LLM.

3) Language backbone layer

Concerns include:

  • What kind of LLM bottom mold is used
  • How big is the context window
  • Whether it is friendly to tool calls, long contexts, multi-round dialogue

No matter how strong the visual ability is, if the language backbone is weak, multiple rounds of question and answer and complex reasoning will still pull the hips.

4) Input tissue layer

Concerns include:

  • Single picture or multi-picture ability is strong
  • Whether to support high-resolution slicing
  • Will the image token explode?
  • How to organize the stitching sequence of text and images

This directly determines the throughput and video memory cost during deployment.

5) Training and task offset layers

Concerns include:

  • Is the data partial Caption, partial VQA, or partial OCR / document / chart
  • Whether there are multiple rounds of multimodal instruction data
  • Have you done special enhancements, such as tables, GUI, mathematical images

This layer is often the easiest to ignore, but the actual best decision is "whether the model is suitable for your business".


3. LLaVA route: Why has it become the starting point for many people to understand VLM?

If which route is the most suitable for getting started and understanding VLM, it is basically LLaVA.

Because its design is very typical and very engineered:

  • Use the already trained Vision Encoder
  • Received to LLM with a relatively simple projector
  • Do multimodal directive fine-tuning

The value of this route is:

It brings the "multimodal ability to the ready-made LLM" thing, which is clear and practical enough.

1) The core idea of LLaVA: try to change the structure as little as possible, and quickly connect the visual ability into the text model

The typical approach of LLaVA can be summarized into three steps:

  1. Turn images into visual features with Clip-like visual encoder
  2. Map visual features to LLM Hidden Size with MLP Projector
  3. Send these visual tokens with the text prompt into the language model

Its advantages are very clear:

  • simple structure
  • easy to reproduce
  • Relatively controllable training costs
  • Great for open source community expansion

This is why the LLaVA system model ecology is particularly active.

2) Advantages of the LLaVA route: low threshold to build, high cost performance for general graphic questions and answers

If your goal is:

  • First, do a working graphic dialogue system
  • Single-picture question-and-answer
  • Do not pursue extreme OCR and document precision
  • Want to train and reason links as simple as possible

The LLaVA route is suitable.

It is usually enough in these scenarios:

  • General picture question and answer
  • scene description
  • Simple graphic reasoning
  • Basic Education Demo
  • multimodal chat prototype

3) The shortcomings of the LLaVA route: high resolution, document details and complex multi-image tasks are often not strong points

Its problem is also obvious.

Due to many LLaVA style models:

  • Visual input resolution is relatively conservative
  • The connection is lightweight
  • Training partial general dialogue

So once the task becomes:

  • Small Letters OCR
  • High-resolution poster analysis
  • Multipage document understanding
  • Comparison of multi-picture details
  • GUI element positioning

Performance often begins to fall off.

This is not to say that LLaVA will not work, but its design goal is even more biased:

In exchange for the low structural complexity, in exchange for strong enough general multimodal ability.

4) How to view the LLLAVA route from the deployment perspective

From a deployment perspective, the advantages of the LLaVA route are:

  • Pipeline clear
  • Engineering stack matures
  • community support wide
  • Easy to connect with inference frameworks such as vLLM, SGLang, Transformers

But you have to pay attention to a few parameters:

  • Image resolution upper limit
  • The number of visual tokens mapped by each graph
  • Additional video memory footprint for Projector
  • Prefill cost in multi-picture scenarios

If you want to do low-cost prototypes or API-level graphic chat, the LLaVA route is worthy of priority.


4. Qwen-VL route: Why is it more concerned about the adaptation of Chinese and multimodal dialogues and complex tasks?

The reason why the Qwen-VL line is highly discussed is that it is not simply pursuing "can see pictures", but emphasizes more:

  • Graphic dialogue experience
  • Chinese Scenario Availability
  • Complex task coverage
  • System-level multimodal extension capabilities

1) The appeal of Qwen-VL is not just vision, but the overall balance of "language base + multimodal ability"

In many cases, the sense of real use is not determined by the front end of the vision alone.

What the user finally feels is:

  • Check if the picture is accurate
  • Ask if it is natural
  • Is the Chinese explanation smooth?
  • Is the multi-round interaction stable?
  • Whether the reasoning process is coherent

The Qwen series itself is relatively strong in language ability, so once the multimodal is firmly connected, the overall experience is often more complete than the model of "the visual is OK but the language backbone is average".

2) Qwen-VL class routes usually pay more attention to multitasking, not just a caption of a graph

In many practical assessments, the Qwen-VL system model is more often used to test:

  • OCR
  • graphic understanding
  • Documentation Questions and Answers
  • Analysis of Chinese picture content
  • Multimodal Conversation

This shows that its design goal is more inclined to the "practical universal Multimodal assistant".

3) The engineering meaning of Qwen-VL: it is easier to use as a general product base, but it is also a test of reasoning resource management

If you use models like Qwen-VL for online services, you usually need to focus on:

  • Video memory curve at high resolution input
  • Token growth caused by OCR/document class images
  • Context length after multiple rounds of history superimposed
  • Throughput attenuation when the image and text are input alternately

In other words, it is often more "all-powerful", but it is also easier to fill up resources in production.

4) Which scenarios are more suitable for Qwen-VL?

If your business is more biased:

  • Chinese Graphic Questions and Answers
  • Documentation and screenshot understanding
  • Needs strong language interpretation skills
  • It is hoped that the model will take into account the chat, OCR and reasoning

That is usually more attractive than the pure LLaVA style route.

But the premise is that you are willing to accept more complex deployment optimizations.


5. MiniCPM-V route: Why does it often appear in the discussion of "lightweight but Surprisingly strong"?

MiniCPM-V is very representative because it embodies a very realistic engineering goal:

It is not blindly expanding, but under the limited parameters and limited resources, the multimodal ability can be as high as possible.

1) The core value of MiniCPM-V: more emphasis on efficiency, compactness and end-to-end friendliness

Many teams don't need an oversized multimodal bottom mold.

What they are more concerned about is:

  • Can a single card run?
  • Is it possible to deploy mobile or edge devices
  • Delay is acceptable
  • Can OCR and graphic Q&A under the small model be played?

The reason why MiniCPM-V is concerned is that it can often give good results under such constraints.

2) Why is the lightweight model still competitive?

Because the performance of the multimodal system is not only determined by the quantity of parameters.

If a model is optimized better in these places:

  • Visual token compression is more effective
  • Input resolution strategy is smarter
  • The distribution of training data is closer to the target task
  • Higher command data quality

It is entirely possible that it can use fewer parameters in exchange for better unit cost performance.

3) Typical applicable scenarios of MiniCPM-V

It is more suitable for:

  • Resource-constrained deployment
  • Local or end-side inference
  • Small and medium-sized graphic services
  • Products that are sensitive to the cost of each reasoning

Of course, the price also exists.

In very complex scenarios, such as:

  • ultra-long document
  • A large number of multi-picture joint reasoning
  • Understanding of difficult cross-regional relationships

Lightweight routes usually still encounter capacities.

4) How to look at MiniCPM-V on deployment

If you value "the highest possible availability under the unit GPU budget", the miniICPM-V line is worth studying.

You need to focus on the observation:

  • The performance cannot be dropped after 4bit / 8bit quantization
  • Where is the balance point of OCR accuracy and throughput
  • Delay changes in single, dual, and multi-picture scenarios
  • Stability under small batch and high concurrency

It's not necessarily the "absolutely strongest" route, but it's probably one of the "most cost-effective" routes.


6. InternVL A type of route: Why are they often tied to high resolution, multi-picture, and document capabilities?

In addition to LLaVA, Qwen-VL, and MiniCPM-V, another direction that is very worth looking at is that the InternVL category, which emphasizes the upper limit of the visual ability, is more important.

The keywords that are often discussed are:

  • High-resolution understanding
  • Multi-map ability
  • Documentation and OCR
  • Stronger Vision Backbone
  • more complex input organization

1) The core goal of this type of route: don't lose visual details too early

The problem with many simple multimodal systems is:

  • The picture is resize, the details are gone
  • Once the patch is roughened, the text will be blurred
  • As soon as the visual token is compressed, the small target and local structure are swallowed

Therefore, it is more important to emphasize the route of high resolution and document ability, and will find a way to preserve more details.

Typical practices include:

  • Dynamic resolution input
  • multi-tile organization
  • Stronger Vision Encoder
  • Special training for OCR / Documentation / Chart

2) Advantages of this type of route: the details and tasks are more stable

If you are testing these tasks:

  • Page-level document Q&A
  • small print recognition
  • long graph analysis
  • Table and chart reading
  • Understanding of multi-picture splicing

Such models are usually more stable than simple single-graph low-resolution routes.

3) The price is also very direct: token, video memory, and delay will be heavier

This is a reality that all high-resolution routes cannot be circumvented.

Because as long as you keep more visual details, you will almost certainly bring:

  • More visual tokens
  • heavier prefill
  • Higher memory usage
  • Worse multiconcurrency ability

So this model is usually more suitable for:

  • high value task
  • Business scenarios with high precision requirements
  • A system that is willing to pay more computing power for image understanding quality

It is not necessarily suitable for the kind of light chat service with high price and high concurrency.


7. Why are some models good at general dialogue, and some models are good at documents/OCR?

This issue is critical.

Many people can easily mistake it as:

The understanding of documents, OCR, and charts is just the result of "a little stronger viewing ability".

Actually not.

They are often different tasks in training and structure.

1) Universal dialogue is more dependent on language trunk and multimodal instruction following capabilities

If the task is mainly:

  • image description
  • Open Q&A
  • chat interaction
  • Multiple rounds of questions

Then the language backbone quality and multimodal instruction data will be very critical.

In such scenarios, the language expression is natural, the logic is clear, and the questioning is stable, which is often more important than OCR's every word.

2) OCR / document is more dependent on high-resolution visual input and special training

If the task becomes:

  • Identify small characters
  • Reading bills, contracts, courseware
  • Understanding Tables and Complex Typography
  • Judgment of buttons, fields, and locations in the screenshots

The key to the decision becomes:

  • Whether the visual details are preserved
  • Have you done OCR/DocVQA/ChartQA class training
  • Whether to support the resolution and input organization that is more suitable for documents

3) So "who is stronger" is a task-related issue

If you're doing a product map Q&A, maybe the LLaVA style model is enough.

If you are doing Chinese invoices, PPT, paper screenshots and table Q&A, then you should give priority to more complex graphic tasks such as Qwen-VL and InternVL.

If you're doing the end-side assistant, then MiniCPM-V may be more suitable than the "big and complete" route.

This is why multimodal selection cannot only look at a total list.


8. From the perspective of parameters: which configurations should these models focus on?

If you are ready to deploy or fine-tuning mainstream VLM, I recommend focusing on the following parameters.

1) Image resolution

This is the parameter that most directly affects the visual details and cost.

You have to see:

  • What is the default input resolution
  • Whether to support dynamic resolution
  • Whether multi-tile is supported
  • Whether the token growth is linearly controllable after the resolution increases

2) Number of visual tokens

Many papers will not make it very straightforward, but they must be counted in engineering.

Because it directly affects:

  • prefill delay
  • Memory occupancy
  • Multi-map ability
  • context window pressure

3) Projector / Connector scale

This affects:

  • Visual Information Transmission Capability
  • Modal alignment quality
  • fine-tuning difficulty
  • training stability

If the Connector is too weak, the model is easy to "see the approximate, but not the details".

4) Context length

The context of the multimodal system is often not just a text window.

You have to count these together:

  • image token
  • Intermediate text generated by OCR
  • User history rounds
  • Model output length

Many online accidents are essentially the context budget is not clear.

5) Quantization of bit width and inference framework compatibility

For example:

  • FP16
  • BF16
  • INT8
  • 4bit AWQ / GPTQ / GGUF

You have to look at quantization:

  • Is OCR obvious drop point?
  • Image detail understanding is deteriorating
  • Does the Inference Framework support visual modules?
  • Is the KV cache and multimodal preprocessing compatible?

Quantization of VLMs is generally more sensitive than plain text LLMs.

Once the visual error is enlarged, it is easy to directly affect the final answer.


9. If I were asked to give suggestions according to the scene, how would I choose?

Here I give an engineering rough division method.

1) If you want to do low-threshold general graphic chat prototype

Prioritize:

  • LLaVA route
  • Community mature lightweight multimodal model

The reason is:

  • Good pick
  • Many information
  • Inference Stack Maturity
  • Enough to do demo and prototype

2) If you want to be a Chinese graphic assistant, screenshot question and answer, document analysis

Prioritize:

  • Qwen-VL system
  • InternVL a model of more complex visual tasks

The reason is:

  • The overall balance of language and multimodal is better
  • Usually friendly to Chinese and complex tasks
  • More suitable for real product interaction

3) If you want to do resource restricted deployment

Prioritize:

  • Lightweight and efficient routes like MiniCPM-V

The reason is:

  • Better unit cost
  • More suitable for on-premises/edge deployments
  • Easier to land on limited GPUs or end-side resources

4) If you want to do high-precision documents, OCR, multi-picture high-resolution tasks

Prioritize:

  • More emphasis on high-resolution input and document capabilities
  • A model that handles tile / dynamic resolution

The reason is:

  • Essentially, this kind of task eats visual details
  • Simple single graph low-resolution model is easy to be unstable

In other words, don't ask "who is the most popular in the community".

And to ask:

What your input looks like, what the task looks like, what your budget looks like.


10. When deploying and landing, the most common pits of mainstream VLM

1) Just look at the Benchmark, not the real input distribution

Many models are very strong on public Benchmark, but your business pictures may be:

  • Phone screenshot
  • long page
  • Excel export diagram
  • blurry photography
  • Compression with watermark

At this time, the public score may not necessarily represent the real effect.

2) Only look at the amount of parameters, not the cost of visual token

Sometimes the small model is equipped with high-resolution multi-map input, and the actual cost is not lower than that of the medium model.

Because the visual token is one of the swallowing killers.

3) Ignore the consistency of image preprocessing

For example:

  • different ways of resize
  • length-width ratio stretch
  • tile order inconsistent
  • Image normalization processing is not uniform

These can make the online effect fluctuate very much.

4) Put the plain text inference experience directly on the multimodal

Some optimization ideas commonly used in plain text LLM may not be directly applicable to VLM.

Because you have to consider additionally:

  • Visual pretreatment time-consuming
  • image token injection method
  • multimodal cache behavior
  • The effect of quantization on visual modules

Therefore, the VLM deployment must be Profiling separately.


11. Final summary

The difference between mainstream VLM models, on the surface, is "different names, different bottom models, and different parameters".

But from the engineering perspective, the ones that really decide to experience are the following groups:

  • Visual Backbone is not strong
  • How to control the image resolution and visual token budget
  • Is Modal Bridge Module Lightweight First or Expression Priority
  • The training data is partial to general dialogue, or partial OCR / document / high resolution task
  • Will the model run as a hosted service or on a resource-constrained device?

If summed up in one sentence:

LLaVA offers a straightforward, engineer-friendly way to connect vision and language. Qwen-VL takes a general-purpose assistant approach that balances language and multimodal capabilities. MiniCPM-V emphasizes what each unit of cost can buy when resources are tight. Models such as InternVL push further on fine visual detail and demanding tasks that combine images and text.

So don't ask "who is the strongest" in general.

More should be asked:

  • Is your task general graphic chat, or document/OCR?
  • Is your deployment budget single card, local, or multi-card service?
  • Are you most afraid of delay, video memory, or unstable accuracy?

Think clearly about these problems, and model selection will be much easier.

I will continue to write a more specific layer in the next article:

How to do multi-picture input, high-resolution understanding and long-graphic scenes? Why do many VLM single pictures are ok, and they start to fall behind as soon as there are multiple pictures and long pictures?

The next article will look closely at image resolution, tiling, and multi-image inputs, why visual token counts grow so quickly, and how to manage that growth in deployment.