In the previous article, we just took the RAG link apart.

If the RAG solves the problem, "how to look at the external data before the model answers", then starting today, the series will officially cut into another very critical main line:

How does the language model start to "see pictures"?

When many people first come into contact with VLM or MLLM, there is a very natural intuition:

  • LLM is responsible for understanding text
  • add an image encoder
  • Feed the image information
  • The model can look at the picture and talk

The direction of this understanding is not wrong, but it is too thick.

As soon as it falls to the model structure, training process and deployment engineering, the problem will immediately become more specific:

  • The image is originally a pixel, and the language model eats token. How to connect it in the middle?
  • Why do visual encoders mostly use ViT/CLIP structures instead of directly throwing the original image to LLM?
  • What exactly is a projector, and why does it exist so much in many multimodal structures?
  • How many visual tokens are the image coded into? What is the control of this quantity?
  • What is the difference between cross-attention, MLP projector, Q-Former?
  • Why is the image resolution 1 liter, and the video memory and delay are easy to explode?
  • What exactly is "aligning" during training? Is it enough to pair graphic and text, or do you need multimodal instruction data?

Behind these questions, they all point to one thing:

Giving an LLM “eyes” takes more than adding an image input. A VLM needs mechanisms alongside the language backbone to represent visual information, align it with language, and feed it into the model.

In this article, I want to make this link clear from the engineering perspective.

I will focus on six things:

  1. Why can't the image be fed directly to LLM, what is the visual encoder doing?
  2. How does the visual token come from, why does patch size and resolution directly affect the cost
  3. What problem does the projector or adapter solve?
  4. Mainstream image access routes: what scenarios are suitable for MLP projection, cross-attention, and Q-Former
  5. What exactly is the "alignment" of training, and why is it not enough to have pictures and texts?
  6. Deployment pitfalls: too many visual tokens, rising latency and inconsistent image–text alignment

If you’re studying LLaVA, Qwen-VL, InternVL, or MiniCPM-V, or preparing to use multimodal capabilities in an application, this article is a starting point for understanding VLM architecture.


First, let's talk about the conclusion: the image is not "directly understood by the language model", but is first turned into a discrete or continuous representation that can be consumed by the language model.

Many introductory introductions will talk about VLM as:

  • image input
  • Model understanding
  • text output

But this hides one of the most critical facts:

The language model itself does not naturally understand pixels.

LLM works because it handles the token sequence.

The original image is:

  • 2D pixel mesh composed of width and height
  • Each pixel contains RGB or other channel values
  • Strong space structure, high redundancy
  • The statistical structure of the text token is completely different

Therefore, if the image wants to "access" the language model, at least one layer of conversion must be completed first:

Compress the visual information in the pixel space into a representation that can continue to be processed by a certain language model.

This conversion is generally done by the visual encoder.

1) The visual encoder is not a simple mention of features, but is doing "the front-end semantic compression of the visual world"

Its work is not just shrinking the graph, but turning the original image into:

  • more compact representation
  • Representation with semantic structure
  • Representation of further alignment with the language side

You can compare it to:

  • tokenizer cut the text into token
  • Vision Encoder cuts the image into patch and encodes it into visual embedding

Of course, these two processes are not exactly the same, but are somewhat similar in system roles:

They are all turning the original modalities into units that are easier to handle in the subsequent backbone network.

2) Why can't I directly send the entire picture flatten and then send it to LLM?

The reason is realistic:

  • The original pixel dimensionality is too high
  • The visual local structure is very strong, and the direct flattening will destroy the space inductive bias
  • LLM is not naturally friendly to 2D local receptive fields
  • Training cost and sequence length will directly get out of control

Therefore, the Multimodal system in reality almost always let the visual side do a round of codes by itself.

This is also why VLM rarely grows directly from the "pure LLM backbone", but is usually made by:

  • a visual encoder
  • a modal connection module
  • a language model backbone

three parts.


2. Why are the visual encoders mostly ViT/CLIP routes?

In today's mainstream VLM, the most common visual front-ends are basically inescapable:

  • ViT
  • CLIP Vision Encoder
  • EVA/SigLIP/InternViT et al. visual backbone

Although they have different training goals, the roles they play in engineering are very close:

First cut the image into a patch, and then encode the patch into a string of visual tokens or visual features.

1) The basic idea of ViT is to process the image as a sequence

This is also the reason why it is easier to match with LLM.

Assuming a graph resolution is:

  • 224 × 224

If the patch size is:

  • 14 × 14

Then the image will be cut into:

(224/14)×(224/14)=16×16=256(224 / 14) \times (224 / 14) = 16 \times 16 = 256

patches.

Each patch is then mapped into a vector, and the encoding continues through the Transformer.

So the image has changed from a two-dimensional pixel mesh to a visual sequence with a length of about 256.

This step is very critical.

Because from here, the visual side and the language side are at least "serialized" to become compatible.

2) Why do many multimodal models like to reuse CLIP Vision Encoder directly?

Because the CLIP series of visual encoders have already pre-trained the data on large-scale graphic and text.

This means it naturally has several advantages:

  • Visual characteristics are more semantically strong
  • There is already a certain correspondence between the graphics space
  • You don't need to learn from scratch "what can be said in images"
  • Can significantly reduce the training cost of the multimodal system

So a model like many LLaVA routes is essentially:

  • Take a trained vision encoder
  • Then try to connect its output to LLM

This is much more realistic than completely end-to-end retraining from pixels to language.

3) The selection of the visual encoder will directly affect the accuracy, speed and video memory

Here are a few common parameters that are very critical:

  • Backbone Scale: B/L/H level
  • hidden size
  • patch size
  • input resolution
  • Whether to support high resolution or dynamic resolution

These parameters are not "decorations in the model paper", but are very directly influenced by:

  • number of visual tokens
  • AT Inference time extension of the single picture
  • Video memory consumption
  • Can the multi-picture scene be able to hold it?

So when choosing a visual backbone, you can't just look at the Benchmark score.

In the multimodal system, the visual front end itself is one of the main throughput and cost.


3. How did the visual token come from? Why do Patch Size and Resolution directly determine the system cost?

This is very worthwhile to talk about separately.

Because many people watch VLM for the first time, they only pay attention to "it can see pictures", but ignore:

Once the image enters the sequence model, the cost is often calculated by token.

1) The higher the image resolution, the more visual tokens are usually

If the patch size is fixed, the resolution is one liter, and the patch number will go up quickly.

Take the most intuitive example.

When the patch size is fixed to 14 × 14:

  • 224 × 224 → 16 × 16 = 256 Patch
  • 448 × 448 → 32 × 32 = 1024 Patch

In other words, if the side length is doubled, the number of visual tokens will be close to 4 times the original.

The computational overhead of attention is usually not linearly rising to the length of the sequence.

This is why the high-resolution input will suddenly become very heavy when the high-resolution input is added.

2) The smaller the patch size, the richer the local details, but the higher token cost

The patch size can be roughly understood as "how large image areas are covered by each visual token".

  • PATCH small: keep more local details, but token more
  • Patch large: less token, but thicker local information

This brings a typical engineering trade-off:

  • OCR, charts, small objects, fine-grained positioning tasks, usually more high-resolution and finer-grained visual token
  • General view Q&A, coarse-grained semantic description, less extreme requirements for patch fineness

Therefore, the visual resolution parameter of VLM is by no means "the higher the better the better".

It's actually balanced:

  • fine visual understanding
  • token cost
  • Video memory budget
  • delay constraint

3) When multi-picture input, the visual token explosion will be more serious

The single picture is already very expensive, and the multi-picture scene will be more obvious.

Suppose there are 576 visual tokens per figure, and the 4 pictures are:

  • 4 × 576 = 2304 visual token

If you add text input and long output, the entire context will quickly approach the upper limit of the model window, or the cost of prefill will increase significantly.

This is why the abilities of multi-picture, long-picture, and high-definition map look cool on the demo, but as soon as you enter production, you must make a token budget very seriously.


4. What is the projector? Why is it almost impossible to get around in the multimodal model?

If the visual encoder is responsible for turning pictures into visual features, then Projector solves the next step:

How can these visual characteristics be successfully caught by the trunk of the language model?

Because the visual side and the language side are often not naturally compatible.

They may be inconsistent in these areas:

  • Embedding dimensions are different
  • Feature distributions are different
  • different semantic spaces
  • Sequences are organized in different ways

Therefore, there must be an intermediate module, which is responsible for "translating" the visual representation into a form that is easier to consume on the language side.

This type of module is generally referred to as:

  • projector
  • adapter
  • connector
  • bridging module

1) The most basic projector is often one or more layers of MLP

Many early or lightweight routes will be very direct:

  • Take the Vision Encoder output
  • Go through a linear mapping or two layers of MLP
  • Put the dimension to LLM Hidden Size
  • Then put these visual tokens as prefixes or insert sequences into the LLM

The advantages of this method are obvious:

  • Simple
  • small parameter
  • Low training cost
  • Easy to quickly build on existing LLM + Vision Encoder

The reason why LLaVA is popular is largely because this connection method is enough to engineering.

2) Projector is not a pure "dimensional alignment layer", it also assumes the distribution task

Many people think of Projector as "change a shape".

This is not complete.

Its more important role is actually:

The statistical distribution and semantic organization of the visual side are as close as possible to the acceptable area of LLM.

Because the LLM backbone is trained on the text token distribution.

If you force a string of completely unfamiliar distributions into it, LLM will not automatically understand it automatically.

So Projector is also involved in modal alignment in essence.

3) The projector is too weak or too small to connect with complex visual semantics

This is also a problem that many systems encounter in the early days.

If the Connector capacity is too small, common phenomena include:

  • Can do coarse-grained image description
  • But the ability to locate the details is weak
  • Poor understanding of OCR, chart, cross-regional relationship
  • Poor picture consistency

That is to say, no matter how strong the visual encoder is, if the middle bridge is too narrow, it is still difficult for the language trunk to really eat enough information.


5. What are the mainstream routes of image access LLM?

From a structural point of view, the mainstream VLM can roughly be classified into several typical routes.

1) MLP Projector route: simple, lightweight, engineering-friendly

This is the easiest category to understand.

The basic idea is:

  • Vision Encoder outputs a string of visual features
  • Map to LLM Hidden Size via MLP/Linear Projector
  • Use these features as a visual token to access the language model

Advantages:

  • simple to implement
  • Low training cost
  • Great for fast multimodal extensions on strong LLMs

Disadvantages:

  • Limited ability to express complex visual details
  • Depend on Vision Encoder itself is strong enough
  • The modeling ability of high-level graphic interaction is not necessarily the best

This route is perfect for:

  • In the project, the multimodal ability is first connected
  • Pursuing lower training thresholds
  • There is already a strong text backbone, I hope to quickly increase the image understanding interface

2) Cross-attention route: make the language side and the visual side do more explicit interaction

Another approach lets the language model attend directly to visual information as it generates text, going beyond simply feeding it visual tokens as a prefix.

This practice is generally more flexible, as it allows:

  • Text Token dynamically selects which visual areas to focus on when decoding
  • Visual and language representations do deeper interactions

Advantages:

  • Graphic interaction modeling ability is stronger
  • More potential for complex relationships and fine-grained understanding

Disadvantages:

  • more complex structure
  • Training and reasoning overhead is usually higher
  • The engineering complexity is also higher

3) Q-Former / query-based route: first use a small group of query vectors to extract key information from the visual side

The idea of this route is:

Don't put all visual patches directly to LLM, but first use a set of Learnable Queries to extract a more compact representation from the visual features.

The core motivation of this is realistic:

  • Too many original vision tokens
  • It is impossible to directly send everything directly into the language trunk
  • Need to do a layer of information compression and selection first

Advantages:

  • Can compress the number of tokens while maintaining the amount of information
  • Friendly for large-scale multimodal pre-training
  • More suitable for visual information abstract injection

Disadvantages:

  • The structure is heavier
  • Higher training requirements
  • The intermediate extraction layer itself also introduces the design complexity

Therefore, from an engineering perspective, these three routes are not absolutely more advanced, but:

How much structural complexity you are willing to use, how much graphic interaction ability and token efficiency are you willing to change.


6. What is the "graphical alignment" mentioned in the training?

This is the easiest sentence in VLM to be said to be empty.

When many articles mention multimodal training, they will say "do graphic alignment".

But if you don't take it apart, it's easy to listen to or don't know what the model has learned.

More specifically, graphic alignment involves at least three layers:

1) Semantic alignment: image content and language description should fall into a similar semantic space

The most basic is:

  • When you see a cat, the text "cat" can't be too far away
  • Seeing the chart, the model should be able to establish the correspondence between the image structure and the language description

The CLIP class training is largely doing this.

It allows the representation of the visual encoder to have the semantic nature of "can and text correspond to each other".

2) Interface alignment: visual representation to become the input that can really consume LLM

This step is often undertaken by Projector / Adapter.

That is to say, even if the visual features already have semantics, if its distribution is completely incompatible with LLM, the language backbone is still difficult to use stably.

So it is not just "graphic matching" during training, but also learning:

  • How to connect visual representation to language trunk
  • After connecting, how does LLM use it when it is generated

3) Task alignment: The model not only needs to "know what is in the figure", but also "know how to answer by task"

This is why pure graphic and text are usually not enough for pre-training.

Because in the real Multimodal interaction, users are not only concerned with caption.

They will ask:

  • Which area in the picture is abnormal?
  • What is the difference between these two pictures?
  • What is the price of the third line on the menu?
  • What does chart trends show?
  • Where is the next step in this screenshot?

These are all tasks, multi-round, and instruction.

So modern VLM generally also needs:

  • multimodal directive fine-tuning
  • OCR/Grounding/Chart/Document and other special data
  • Sometimes it also adds thinking chain, multistep, multi-picture data

In other words:

Graphic alignment allows the model to "connect the graph and words", and the multimodal instruction is trained to let the model "know how to use this connection around the user's task".


7. Why do many VLMs seem to "look at pictures", but once the details are reached, the task is unstable?

This is a particularly common gap in the Multimodal system.

The demo phase looks good, but as soon as the complex task is on, the problem begins to expose.

There are usually these types of reasons.

1) The visual token is too thick and the local information is not enough

If the resolution is low and the patch is large, the model is easy to be unstable for small targets, text, small tables, and local layouts.

2) The capacity of the projector is limited, and the complex visual relationship cannot be passed

If the bridge layer is too narrow, many higher-order semantics have been lost before accessing LLM.

3) The training data is partial to Caption style, and the task distribution is too narrow

If the model has mainly seen:

  • "Describe this picture"

Rarely seen:

  • OCR Q&A
  • Comparison of multiple pictures
  • grounding
  • document parsing
  • GUI operation understanding

Then it will naturally be better at "speaking", not good at "doing it very accurately".

4) The language model will complement the fluency, but it does not mean that the visual evidence is really enough

This is a bit like a RAG.

Many multimodal hallucinations are essentially the same:

  • The model language side is too good at speaking
  • The visual side evidence is not strong enough
  • In the end, the output becomes "like understand", but in fact, it is filling the hole

So to judge whether a VLM is good enough, you can't just see if it's caption is smooth.

More importantly, look at:

  • OCR is not allowed
  • Grounding unstable
  • How is the consistency of multi-picture?
  • Can high-resolution documents understand?
  • Whether it will stably refer to visual evidence

8. Deployment pitfalls: resolution, visual tokens and GPU memory

This paragraph is important if you are ready to connect the VLM to the real system.

1) The higher the resolution of the visual input, the better

High resolution does often lead to better detailing capabilities.

But the price is also very direct:

  • increased visual token
  • prefill becomes heavier
  • Increased video memory usage
  • lengthy delay
  • Decreased multi-picture concurrency

So the more practical approach in engineering is usually:

  • Different tasks set different resolution gears
  • OCR / Documents / Charts take high resolution paths
  • Ordinary viewing Q&A takes medium and low resolution paths
  • If you can cut it, cut it first, don't come up and hard to feed the whole big picture

2) The image preprocessing strategy will directly affect the online stability

For example:

  • Whether resize is distorted
  • Whether to keep the aspect ratio
  • Do you do tiling
  • Whether the order of input of multiple images is fixed
  • Whether the color space, rotation, and cutting edges are unified

These look like "pretreat little things" and actually strongly affect the model's behavioral consistency.

3) Multi-picture, multipage PDF, long document scenarios should be extremely careful token budget

Many real tasks are not single graphs, but:

  • Multiple product pictures
  • Multipage document
  • An entire contract
  • A series of GUI screenshots

At this time, if you do not:

  • sieves
  • Pagination
  • Partial cropping
  • Phaseed Reasoning

will appear soon:

  • token explosion
  • delay soaring
  • Video memory is not enough
  • The quality of the answer has declined

Therefore, in the multimodal deployment, a very common correct idea is:

Do visual input management first, and then do multimodal generation.

Don't put all the pictures without brains.


9. If I want to make a usable VLM system now, how would I suggest starting?

If you are not doing research papers now, but want to do a multimodal system that works on engineering, I would recommend starting this way.

1) Identify the task type first, rather than superstitious about a model name first

Ask first:

  • Mainly Caption or Q&A?
  • Single or multi-picture?
  • OCR, document, chart, or natural image understanding?
  • Need Grounding?
  • How high is the input resolution?

Because different tasks determine:

  • what vision to choose backbone
  • Whether high-resolution support is required
  • Do you need more graphic skills?
  • Whether to strengthen OCR/DOC understanding

2) First calculate the cost of visual token

Don't just look at the model parameters.

In the Multimodal system, what is really easy to be underestimated is:

  • single figure token number
  • Token number after multi-picture superposition
  • Delay difference in different resolution gears

This will directly determine your concurrent cap and GPU budget.

3) If the business emphasizes accurate positioning, don't just look at the general chat experience

Many model chat experiences are good, but:

  • Bounding / Grounding poor
  • OCR is unstable
  • Table parsing is easy to misplace

Therefore, the evaluation must be close to the task, not just a few questions "what is in the picture".

4) Can be staged in stages, don't end-to-end

For example:

  • Document task: first page filter, then high resolution
  • Multi-picture task: first coarse sieve, then look at the key map
  • GUI task: first locate the area, then do operational reasoning

This type of Pipeline practice is usually more stable and cheaper than "getting all visual content to a model at once".


10. Final summary

The reason why images can "access" the language model is not because LLM has suddenly learned to look at pixels.

Instead, the multimodal system adds several key structures:

  • Visual encoder presses pixels into visual representations
  • Patch and Resolution determine the number of visual tokens and the granularity of the details
  • Projector / Adapter to connect visual features to LLM consumable space
  • Align training allows images and language to start collaborating in the same task framework
  • Instructions fine-tuning and special data let the model learn to use visual information around real problems

So the essence of a VLM is not as simple as "LLM + picture input".

More precisely, it is:

In addition to the trunk of the language model, a complete set of visual perception, modal bridge and task alignment mechanisms have been added.

If you are now selecting or building a multimodal system, I would recommend three things first:

  • Is the cost of visual token clear?
  • Is the Projector / Connection Module enough to support task complexity
  • Whether training and reviews really cover the visual tasks you care about, not just generic caption

These three things are often larger than simply changing a larger bottom model, and more determine whether the system is "really usable" in the end.

Next article I will continue to follow this line:

What is the difference between the structures of LLaVA, Qwen-VL, and MiniCPM-V. Why are some models better at general-purpose conversations, and some more suitable for document, multi-picture or high-resolution scenes?

By then you will see more clearly:

The gap between the multimodal model is often not just the parameter scale, but the combination of visual front-end, connection mode, training data and task objectives.