The previous article compared the architectures behind the main vision-language models.
The differences between LLaVA, Qwen-VL, MiniCPM-V and InternVL help determine which tasks suit each model. In deployment, the following inputs often separate systems more clearly than a question about a single ordinary image:
- Reasoning across multiple images
- Ultra-High Resolution Image Understanding
- Long screenshots, long posters, long document pages
- Forms, bills, paper pages, product manuals
- Tasks that need cross-map comparison, cross-regional positioning, and cross-page summary
When many teams are doing demos, they test a 448×448 ordinary picture, and the effect looks good.
But as soon as you enter the real scene, the problem will be exposed immediately:
- As soon as there are more pictures, the number of visual tokens expands rapidly
- The resolution is high, and the memory and delay are obviously out of control
- OCR details are dense, and the model begins to miss words, serial, and misunderstood parts
- When you need to compare across graphs, the model will mix the information of different pictures together
- When the long map is unreasonable, the model is clearly "seeed", but cannot answer the global relationship
Therefore, the focus of this article is not to talk about "model supports multiple graphs".
I would like to talk about three things from an engineering perspective:
- Why Multi-Graphics and High Resolution Make VLMs Expensive
- How do mainstream models use to control the visual token explosion?
- In the real deployment, how to design the input link and parameters for multi-map, long-map, and high-resolution tasks
If you are doing document comprehension, GUI Agent, chart analysis, inspection identification, bill review or multipage question and answer, this article will be more practical than just looking at Benchmark.
1. First, let’s talk about the conclusion: multi-picture and high-resolution are not as simple as "feed a little more", but in the context, video memory, and delayed grabbing the budget.
Many people will think of multimodal input as a very natural sentence:
Since the model can see a picture, it should also be able to see a lot of pictures.
The direction is correct, but the price is far greater than expected.
Because for most VLMs, pictures will eventually be encoded into visual tokens, and then sent to the backbone of the language model together with the text token.
This means that as long as the number of pictures goes up and the resolution increases, the system cost will increase simultaneously.
The cost is mainly reflected in four places:
1) The number of visual tokens has skyrocketed
Assuming that the visual encoder uses the ViT route, the patch size is 14, then the higher the input resolution, the more patches will be cut out.
A rough intuition is:
- 448×448 is about 4 times more than the 224×224 patch
- 896 × 896 further up, the number of visual tokens will continue to increase the square level
- If it is still multi-picture input, the cost will continue to stack linearly
Therefore, high-resolution and multi-picture superposition are the easiest combinations to top out the reasoning cost.
2) The LLM context budget is eaten by visual input
Text models are not infinite contexts.
Even if the backbone supports long contexts, after the visual token enters, it will still compete with the text instruction, historical dialogue, and retrieval context.
The result is:
- When there are too many figures, the space that can be left for text reasoning becomes less
- In multiple rounds of dialogue, the old image information may be truncated
- In the mixed scene of RAG + multi-picture, it is easy to appear "The picture is stuck in, but the text data cannot be stuffed"
3) Prefill delay significantly increases
Many people only stare at the speed of decode, but in the multimodal scene, prefill is often more painful.
Because a large number of visual tokens will fill the model at one time in the input stage, directly pulling up the first word time (TTFT).
Typical phenomena are:
- The question and answer of the single picture is still smooth
- The first response becomes slower when comparing multiple graphs
- The number of pages in the document is over, and the system is like "stuck"
4) The relationship between images does not automatically learn well
Even if the model supports multi-map input, it does not mean that it is naturally good at:
- Differences between Figure 1 and Figure 2
- Cross-page references in multipage documents
- The logical series connection of the front and rear areas in the long screenshot
- Tracking and comparison of the same objects in multiple figures
In other words, multi-picture is not only a problem of computing power, but also modeling and training.
2. Why is high-resolution tasks so difficult? The difficulty is not just "the picture is clearer", but the details and global information are difficult to keep at the same time
The core contradiction of high-resolution input can be summed up in one sentence:
You want to see both small prints, and you want to keep the global structure of the entire picture.
These two things often fight each other.
For example, in document understanding:
- You need to read the header footer, table cell, footnote, formula, small font
- At the same time, it is also necessary to understand the entire page layout, paragraph level, chart and body correspondence
If the whole picture is directly reduced to low resolution:
- The global structure is preserved
- But some text, legends, and small controls will be lost
If you cut the graph into many high-resolution local blocks:
- Local details are preserved
- But the global layout relationship is easy to lose
So high-resolution understanding is essentially doing a balance:
- global vision: Know what the whole picture is talking about
- partial: see small characters, small icons, small areas
- Computing costs: Can't lift token and video memory infinitely high
This is why many mainstream VLMs will introduce strategies such as dynamic resolution, cut graph, tiling, and multiscale coding.
3. How to solve the mainstream scheme: fixed resolution, cut map, dynamic resolution, and multiscale are the four most common routes
1) Fixed resolution: simple to achieve, but the upper limit is obvious
The most direct way is to uniformly resize all pictures to a fixed size, such as 224, 336, 448, 672 and the like.
Advantages are obvious:
- Input form is stable
- easy to deploy
- Video memory and throughput are better estimated
- Suitable for general graphic questions and answers and ordinary scenes
But the problem is also obvious:
- Small text and dense tables are easy to paste
- Long diagrams, posters, and flow charts are often crushed
- Document class task ceiling is low
So fixed resolution is more suitable for:
- Universal Vision Q&A
- Commodity diagram, natural image understanding
- Light tasks that do not require high OCR accuracy
2) Cut the picture / Tiling: the details are the most direct, but the token is easy to explode
Another approach is to cut high-resolution images into multiple tiles, and then encode them separately.
The benefits of doing this are:
- Details within each tile are clearer
- Friendly OCR, documentation, charts, GUI tasks
- You don't have to compress the whole picture limit to blur
But the price is also very direct:
- The more tiles, the more tokens
- Cross-tile relationships are harder to learn
- Significant increase in inference delay
Therefore, the cut map cannot be opened without brains.
The real project to do with:
- tile upper limit control
- Overlap Zone Control
- tile collation unified
- Joint input of local tile and global thumbnails
3) Dynamic resolution: allocate the budget according to the content complexity
In the past two years, many strong multimodal models will introduce dynamic resolution ideas.
The core logic is not a one-size-fits-all picture, but is determined according to the image content, aspect ratio, and layout complexity:
- How many tiles to use
- How high is the resolution of each tile
- Whether to keep the global image
- What is the upper limit of the total visual token
This practice is more flexible than fixed resolution, and is usually more suitable for real business.
Because the complexity of the image in the real world is very different:
- A natural picture does not require document-level fine processing
- One-page invoice and one-page paper layout complexity are also different
- If a long screenshot is just a chat record, the processing strategy of a dashboard screenshot should also be different
4) Multi-scale coding: look at both global and local
There is also a common idea, which is to send the different scales of the same picture into the system together.
For example:
- A low-resolution global map is responsible for layout perception
- Several high-resolution local blocks are responsible for detailed identification
This can alleviate the problem of "just look at the parts and lose the global situation, only look at the global situation and can't see the details".
It is especially suitable for:
- Documentation
- Table and Chart Analysis
- GUI screenshot understanding
- Maps, posters, long flow charts
However, this kind of method also requires stronger input organization and training data support, otherwise the model is prone to appear:
- Know the local content, but will not summarize the global
- Can describe the layout, but the identification of specific details is unstable
4. How to organize multi-map input? Sequence, grouping, identifier, more important than "can you send multiple pictures"
The most common misunderstanding of multi-picture tasks is to put a few pictures directly into it, and then ask:
"Help me compare these pictures."
This is sometimes possible, but the stability is usually not enough.
Because the model does not necessarily know:
- Which is picture 1 and which is picture 2
- What dimension do users want to compare
- Which graphs are the different time states of the same object
- Which pictures are the different pages of the same document
Therefore, multi-map input must be structured.
1) Explicit numbering for each picture
The most basic and effective way is to write clearly:
- Image A: pre-conference version
- Image B: post-sessional version
- Page 1: Directory Page
- Page 2: Body Page
- Screenshot 1: Before login
- Screenshot 2: After login
This can significantly reduce model mixing.
2) Write the task goal as an alignable comparison dimension
Don't just say "compare".
It should be split into:
- Compare Layout Differences
- Compare numerical changes
- Find a new button
- Compare the conclusions of the two-page document
- Summarize the problems common in the three pictures
Models are most afraid of blurred task goals in multi-picture scenarios.
3) Divide multi-picture into "global group" and "local group"
For complex tasks, it is recommended not to tile all the figures into a list at one time.
A better way is usually:
- First give the global picture/full page diagram
- Give the key partial map/enlargement to the key part
- Finally, the corresponding relationship is explained in prompt
This is more stable than a simple "heap map".
4) Control the number of pictures in a single request
Even if the model claims to support many pictures, it is not recommended to stuff too much at one time in the project.
A more stable approach to experience is:
- First-level screening
- Select candidates
- Do the second-level intensive reading
This is very similar to RAG, which is essentially layered processing, rather than pressing all inputs into the backbone at once.
5. How to understand the key parameters: Image Resolution, Patch Size, Tile number, Max Image tokens, these are the most worthy of attention
If you want to tune the multimodal system, I recommend focusing on the following parameters.
1)image resolution
It determines what size the input image will be processed before entering the visual encoder.
The higher the resolution:
- Local details are easier to retain
- OCR and small target recognition are usually more stable
- However, the computational cost, video memory, and delay are all higher
Practical advice:
- General Q&A can start with medium resolution
- Document/OCR/GUI tasks are gradually improved
- Do not default to the highest resolution for all scenes
2)patch size
The smaller the patch size, the more patches cut out under the same resolution, and the details are more detailed, but the cost of token is higher.
From an engineering point of view, it and the input resolution are multiplication relationships, not independent relationships.
Therefore, you can't just look at the "model supports high resolution", but also look at the patch division granularity of the visual encoder.
3)tile count / dynamic tiles
This is one of the most easily out-of-control parameters in high-resolution tasks.
The more tiles:
- The easier it is to see the parts
- However, delays, video memory, and throughput will significantly deteriorate
In practice, it is best to set the hard cap for tile and set a template for different tasks:
- Normal picture: low tile or single image mode
- Documentation page: Medium Tile
- Super long graph or complex chart: high tile, but with a limiting
4)max image tokens / visual token budget
Many systems eventually get stuck, not because the model is "incomprehensible", but because the visual token budget design is out of control.
It is recommended to treat it as a first-level system parameter, not to hide the details.
Because it directly affects:
- Is it possible to keep enough text context
- The first word time of the request
- GPU video memory peak
- concurrency
6. Deployment actual combat: multi-map and high-resolution tasks, do not directly track the ability of the single model, first dismantle the service link
In practice, when a task involves multiple images, tall images, or high-resolution inputs, breaking the pipeline into stages is often more reliable than asking the strongest available model to handle everything in one pass.
1) Classify first, then intensive reading
For example, first use a lightweight strategy to determine which category the input belongs to:
- Nature
- Documentation page
- form page
- Long screenshot
- Multi-picture comparison task
Then decide whether to enable high-res, multi-tile, or multi-picture modes.
This saves a lot of unnecessary computing power.
2) Take a closer look first, then take a closer look
For long graph or multipage documents, it is recommended to be divided into two stages:
- The first stage: low-cost global scanning, find key areas
- Stage 2: High-resolution local intensive reading
This is closer to real production needs than full-scale high-precision processing.
3) In the graphic and text mixing task, leave a budget for the text
As soon as many systems encounter documents, they pour all the page pictures into the model, but the text description, rules, and retrieval fragments cannot be plugged in.
A more reasonable approach is:
- Make images responsible for layout and visual evidence
- Let OCR / Structured Extract Responsible for High Density Text
- Let LLM be responsible for comprehensive inference and generation
Don't let a model memorize all the links alone.
4) Treat OCR and VLM as complementarity, not alternative
In many document scenarios, I would recommend using VLM and OCR in combination.
The reason is simple:
- OCR is good at high-precision text transcribing
- VLM is good at joint graphic understanding, layout reasoning, and cross-regional relationship explanation
If you eat all text-intensive documents with only VLM, the cost is high and the stability is not necessarily the best.
A more stable link is usually:
- OCR extract text
- Layout analysis and extraction area structure
- VLM handles graphic relationships that are difficult to regularize
- LLM aggregates to generate final answers
7. The most common pit: the system does not die in the model that is not strong enough, but that the input strategy is unreasonable
Here are a few of the pits that I think are the most common.
1) All pictures are processed according to the highest specifications
The consequences are:
- GPU video memory long-term high voltage
- Very poor concurrency
- The first response time is ugly
- cost out of control
The correct way is to enter the classification, and different tasks go through different budget templates.
2) The long picture is only cut to the part, and the global map is not retained
In this way, the model may recognize each block, but do not know the structural relationship of the entire graph.
3) Only keep the global picture, do not zoom in on the part
In this way, the model knows what the graph is talking about, but OCR, small icons, and detailed tables will make frequent errors.
4) There is no number and task constraints for multiple pictures
The result is often:
- The answer is mixed together
- Comparative Dimensional Deviation
- Unstable cross-picture references
5) Use the multimodal model as a universal OCR engine
VLM is very strong, but not all high-density text tasks should be directly hard.
Traditional OCR + rules/structured methods are often more stable and cheaper when the task objective is high Accuracy field extraction.
8. How to choose a plan: decide the multi-map and high-resolution strategy according to the task type, instead of the other way
If you are making a selection, I suggest using the following idea.
1) General graphic questions and answers
Priority target:
- low delay
- medium cost
- stable enough
Suggestion:
- fixed resolution
- Single picture or a small number of more pictures
- Control Vision Token Budget
2) Document understanding / OCR enhancement
Priority target:
- Warranty details
- Look at the page
- Able to understand across regions
Suggestion:
- dynamic resolution
- global graph + local tile
- OCR and VLM combination
3) Multi-picture comparison / patrol / GUI Agent
Priority target:
- Cross-Graphic Consistency
- Timing or state difference identification
- Clear positioning point of change
Suggestion:
- Picture number
- Fixed Compare Dimensions
- Stage screening and intensive reading
4) Long screenshot / long document / multipage report
Priority target:
- keep context
- control token explosion
- Improve the ability to summarize across pages
Suggestion:
- Pagination or segmentation processing
- Rough reading first and then intensive reading
- Don't stuff all pages at once
9. Final summary: multi-picture and high-resolution ability, which determines whether the VLM can really enter the production scene
A single ordinary picture quiz, today is no longer the most distinguished ability of the Multimodal system.
What can really determine whether a VLM is worthy of landing is more and more engineering questions:
- Can multiple image inputs be compared stably?
- Can you see the details clearly without dragging down the cost at high resolution?
- Can long-graph and document pages take into account both the global and local
- Can the visual token budget be balanced with the text context, delay, and concurrency
So look at a multimodal system, don't just look at it "can see the picture".
It also depends on whether it has a set of explicit input organization, token budget and service layering strategies under complex input conditions.
In one sentence:
With multiple images, high resolutions, or long documents that combine text and images, the challenge is deciding which visual details matter enough to spend the limited input budget on.
In the next article, we continue to move forward and enter the training problems that many teams really encounter:
How to do multimodal fine-tuning? How to match the data of the picture and text? How to determine the key parameters of Vision LR, LLM LR, freezing policy?