Cover illustration

Date: 2026-08-09
Suitable for readers: Graduate students who are ready to enter the direction of the Multimodal model, new scientific research, and AI readers who have engineering background, want to systematically read papers and do reappearance.

Abstract

multimodal LLM The paper is easily read as "model name inventory": CLIP, Flamingo, BLIP-2, LLaVA, Qwen-VL, InternVL, Molmo, Gemini, GPT-4o, Llama Vision...Each article is said that the model is stronger, the data is more, and the Benchmark is higher. But the reading method that is really suitable for scientific research training is not to memorize the list, but to ask three questions: How does visual information become a token that can be used in language models? How to change the model behavior of graphics, video, document and instruction data? Is the so-called multimodal reasoning in the examination of perception, OCR, spatial relationship, math chart, long video memory, or language prior?

This article gives a Multimodal LLM paper intensive reading route. It divides the paper into four layers: visual representation, modal connection, alignment and instruction data, evaluation and reasoning. After reading a VLM paper, you should be able to draw its input resolution, visual encoder, connector, LLM backbone, training phase, data recipe, evaluation collection and failure type. When reappearing, don't train the large model at once, but start with open source weight evaluation, data format replication, a small amount of instructions Fine-tuning and error analysis.

Contents

  1. Why multimodal LLM is worth reading
  2. A reading roadmap
  3. Represents the paper timeline
  4. Four key issues: Encoder, Connector, Data, Reasoning
  5. Methods and experiments compared
  6. How to do a reproducible little experiment
  7. Common misconceptions
  8. Research questions for graduate students
  9. References

Why is this topic important

Multimodal LLM is a technical route from "seeing pictures and talking" to "reasoning with vision as evidence". Early Vision-Language Model focuses more on retrieval, classification, Caption, VQA; modern MLLM hopes to process images, videos, documents, charts, screens, math problems and multiple rounds of instructions in the same dialogue interface. The question also becomes harder: Does the model really understand the graph? Is it just using the language cues in the stem and options? Is it wrapping OCR errors into confidential reasoning? Are new hallucinations appearing in charts, medical images, GUI, long videos?

For graduate students, this direction is especially suitable for training three abilities. First, the ability of architecture dismantling: the same is "access vision", some papers are frozen CLIP/ViT, only projectors are trained; some use Q-Former, cross-attention, resampler; some start to do native Multimodal pre-training and multiscale visual token. Second, data awareness: Multimodal capabilities come from data types, filtering methods, synthesis instructions, OCR/Grounding labeling and preference data, rather than a single module innovation. Third, evaluation sensitivity: mmu, MathVista, MMBench, Video-MME, CharXiv, MMStar, etc. Benchmark do not measure the same ability, and directly comparing the total score is often misleading to the topic.

text road map

Core reading method: draw the information flow first, then look at the score

When reading a Multimodal LLM paper, it is recommended to draw a five-segment information stream first:

input layer: Does the paper support single, multi-picture, video, document, screenshot or audio? Is the image cut into patch, multiscale CROP, arbitrary resolution tile? Is the video framed, time and position coding, or is the audio and video interleaved input?

visual encoder layer: Use CLIP ViT, SigLIP, EVA, InternViT, self-developed ViT, or a visual module for end-to-end training? Is the visual encoder freezing or participating in training? If it is frozen, does the paper explain the bottleneck of resolution, OCR, and fine-grained localization?

Modal connection layer: How does the vision token enter LLM? Common routes include linear Projector, MLP Connector, Q-Former, Perceiver Resampler, cross-attention adapter, token pooling, dynamic resolution Fusion with multi-layered features. The focus here is on the number of tokens, because it directly affects the context cost, long video ability and inference delay.

language model layer: backbone is LLaMA/Qwen/Mistral/OLMo/InternLM/Gemma/Phi or closed source model? Does the paper dismantle the language ability and visual ability for evaluation? If the visual model is changed backbone, the benefits come from visual training or the language base is stronger?

Training and Alignment Layers: At least split into pre-training, graphic alignment, visual instruction fine-tuning, preference optimization/safety alignment, visual reasoning enhancement or test-time scaling. Don't just record "how much data you use", but also record the data as caption, OCR, region grounding, multi-image, video QA, chart, document, GUI, or distilled by strong models Synthetic Instruction.

A reading guide to the key papers

The first stage: visual-language representation alignment. CLIP uses large-scale graphic and text comparison learning to prove that natural language supervision can form migrating visual representations. It is not a modern chat style VLM, but it defines the visual encoder starting point for many subsequent models: the image does not become a category first, but an embedding that can be aligned with the language space. When intensively reading the CLIP, you should focus on training goals, negative sample size, zero-shot prompt and robustness gap, not just ImageNet.

The second stage: freeze the large model and learn the modal interface. Flamingo connects the frozen visual model and the frozen language model, and supports interleaved image/video/text through Gated Cross-attention and Perceiver-style resampler FEW-SHOT PROPTING. BLIP-2 uses the Q-Former as a lightweight bridge module to do two-stage pre-training between the frozen image encoder and the frozen LLM. The core problem at this stage is how to cross the modal gap between "visual representation" and "language generation" with fewer trainable parameters.

The third stage: the visual instruction fine-tuning . LLaVA uses visual instruction data generated by GPT-4 to train a large model for dialogue. It shows how strongly a format that pairs an image with a question, an answer, and an explanation can shape a multimodal LLM’s behavior. Later work, including LLaVA-1.5, LLaVA-NeXT, and LLaVA-OneVision, extends support for higher resolutions, multiple images, and video. Read the data construction procedures, training stages, and evaluation scripts closely. Small changes in any of them often produce large swings in scores.

The fourth stage: open source strong model and native multimodal pre-training. Qwen-VL/Qwen2-VL/Qwen2.5-VL/Qwen3-VL, InternVL2/3/3.5, Molmo, Llama 3.2 Models such as Vision, MiniCPM-V/O, NVLM, etc., have advanced from "receiving diagrams" to "high-resolution documents, long video, visual positioning, chart reasoning, GUI operations and multimodal preference optimization". For example, the Qwen2.5-VL technical report covers the scale of 3B, 7B, 72B, and emphasizes document resolution, target positioning and long video understanding; ViT features and visual agent capabilities. InternVL3 puts Native Multimodal Pretraining, Mixed Preference Optimization and Multimodal Test-Time Scaling to the core position. Molmo/PixMo emphasizes open weights, open data and reproducible training formulas.

Stage 5: From the picture understanding to Omni/Video/Action. Reports such as GPT-4o, Gemini 2.x, Qwen2.5-Omni, Mini ICPM-O, etc. show that the cutting-edge model begins processing image, text, audio, video and voice output. For the topic of this article, the focus is not to track all product capabilities, but to understand how multimodal input changes the evaluation method: audio and video synchronization, timestamp, long context, streaming input, real-time interaction and security evaluation will make traditional VQA reproduction experiments not enough.

Intensive reading of key papers: four entrances

1. Vision Encoder: What exactly does it see?

Many papers regard the visual encoder as a "off-the-shelf module", but the upper limit is often determined here when it is reproduced. You want to record the encoder type, input size, patch size, whether it is multiscale, whether any resolution is supported, and whether it is OCR/document optimization. The model card of the Llama 3.2 Vision clearly states that its visual model expresses the image encoder representation to the Llama 3.1 text model through the individually trained Vision Adapter, and gives 11B, 90B Scale of two visual models. Molmo's official blog shows that its model uses multiscale, multi-crop preprocessing, ViT image encoder, MLP Connector and Token Pooling. Such details are more valuable for research than "models are strong".

When reading the encoder part, you can ask: If the image is reduced, what will the text, table, and small targets be lost? How to keep the spatial relationship if the tile is cut? If the video only draws a small number of frames, can the sequence of events still be reasoned? If using Frozen CLIP, is visual characterization suitable for documents, medical images, GUI and geometry?

2. Connector: Where is the modal gap being compressed?

Connector is the most easily underestimated part of Multimodal papers. Linear Projector is simple, inexpensive, and easy to reproduce; Q-Former and resamper can compress the visual token and select query; cross-attention adapter It allows the language layer to read visual information on demand; multi-layer feature fusion tries to preserve low-level details and high-level semantics at the same time. The Qwen3-VL technical report mentions that DeepStack integrates multiple layers of ViT features, which is the representative signal of "do not only take the last layer of visual features".

When intensive reading, don't just look at the module diagram, check whether the training target really forces the Connector to learn Grounding. Only doing Caption may make the model good at description, not necessarily good at positioning; only doing VQA may overfitting questions; when OCR and structured document data are lacking, charts and paper screenshots are often short-sized.

3. Align data: what samples do the capabilities come from?

The data of Multimodal LLM is usually mixed graphic pairs, caption, OCR, region-level grounding, multi-round visual dialogue, video subtitles, document quiz, math charts, GUI operations and model generation. Synthetic Instruction. We must be especially alert to the "data scale worship". Larger data is not automatically equal to better Reasoning; data coverage, labeling quality, deduplication, visual dependence of the topic and evaluation of pollution are more critical.

The value of Molmo/PixMo is to put open data and training recipes in the research question center. The value of other technical reports lies in how complex the engineering training stage is exposed. When reading these papers, you should write "where the data comes from, whether it is open, whether it can be re-engraved, whether the distillation closed source model, and whether it overlaps with the benchmark" into notes.

4. Multimodal reasoning: Is it real reasoning, or is it a language prior?

MMMU emphasizes university-level multidisciplinary knowledge and heterogeneous image types; MathVista focuses on mathematical reasoning in visual context; MMMEnch emphasizes multidimensional ability and circular variability; Video-MME Pay attention to the understanding of long and short videos; CharXiv uses real paper charts to test Chart understanding; MMStar clearly points out that many samples can be answered without visual input, and propose multimodal gain and leakage indicators. The common reminder of these benchmarks is that the multimodal score is not a single ability.

When doing the essay intensive reading, it is recommended to classify each Benchmark into a type of problem: perception/OCR, spatial relationship, chart, mathematics, document, video time, domain knowledge, hallucination diagnosis, and language leakage. Then look at which classes the model improves on. If only papers with average scores are reported, the intensity of evidence is not enough.

Methods and experiments compared

Route Representative paper/system core mechanism Focus on intensive reading Reproduction Entrance
Graphic comparison and characterization CLIP Graphic and text comparison learning, zero-shot classification Training goals, prompts, robustness Run small data with open_clip zero-shot/linear probe
Freeze model bridging Flamingo、BLIP-2 resampler、cross-attention、Q-Former Freeze policy, visual token compression Reproducing small VQA or caption fine-tuning
visual instruction fine-tuning LLaVA series projector + visual instruction tuning Instruction data, two-stage training, evaluation scripts Run LMMs-Eval/VLMEvalKit with open source weights
Open Source Strong VLM Qwen-VL、InternVL、Molmo、MiniCPM-V High resolution, multi-picture, video, document, preference optimization Data Recipes, OCR/Grounding, Token Budget Fixed model, comparison resolution, prompt, sampling and evaluation
omni / video / agentic GPT-4o、Gemini、Qwen-Omni、Qwen3-VL Audio and video, long context, time alignment, GUI/Agent Input Protocol, Time Modeling, Safety Evaluation Controlled case studies with open API or open source model

How to do a reproducible little experiment

Do not start with training 70B VLM. A reasonable postgraduate reproduction experiment can be designed in this way:

  1. Choose an open-source model family such as Qwen2.5-VL, LLaVA-OneVision, InternVL, or Molmo. Pin the exact checkpoint and the versions of transformers, vLLM, and VLMEvalKit.
  2. Choose two subsets of benchMarks with complementary capabilities, such as MathVista Testmini + CharXiv Sample, or MMBench dev + video-mm short subset. If the ranking or data version changes, record "waiting for manual verification".
  3. Create strict logs: model name, commit, weight source, image resolution, max pixels, frame number, prompt template, temperature, seed, hardware, failure example.
  4. Make a small ablation: only change one variable, such as image resolution, whether the OCR text is external, whether multiple images are merged, whether CoT Prompt, whether to join self-consistency.
  5. Outputs incorrectly categorized, rather than just outputting the total score. At least it is divided into OCR errors, positioning errors, relationship errors, mathematical reasoning errors, language priors covering visual evidence, formatting errors, and abstention/security policy impact.

The value of this experiment is to train you to split the "model ability" into observable phenomena. Even if it does not surpass any SOTA in the end, it can form a reliable reproduction notes and follow-up ideas.

Common misconceptions

Misunderstanding 1: Treat VLM as LLM and add pictures. The real problem is how the visual token is compressed, aligned, and retrieved. Images are not Prompt attachments, but another set of evidence with noise, resolution and spatial structural limitations.

Misunderstanding 2: Only compare the total score of the list. Multimodal Benchmark has a big difference in the question type. MMStar has reminded: some topics may have visually non-essential or leaking problems. When reading a paper, you must check the baseline, text-only baseline and topic deduplication strategy.

Myth 3: Ignore data openness. If training data, filtering rules, synthetic instruction generators and review scripts are not available, the paper can still learn ideas, but the reproduction target should be downgraded to "behavioral reproduction" or "ablation" "Reproduction", cannot promise complete training reappearance.

Myth 4: The default CoT improvement represents the improvement of visual reasoning. The long explanation in the visual problem may just be the complement of the language model. It is necessary to check whether the intermediate steps can be supported by image evidence, and if necessary, do occlusion, crop, no pictures, counterfactual images and options to disrupt the experiment.

Research questions for graduate students

  1. Visual Dependence Evaluation Cleaning: Screen out samples that can be correct without a picture from the existing Benchmark, and construct a more reliable subset of visual-required.
  2. Connector Layer Ablation: Compare the token efficiency and error types of Projector, Q-Former, Resampler, and Multi-Layer Feature Fusion under the same Encoder and LLM.
  3. Fine-grained error analysis of documents and charts: Analyze chart elements, coordinates, legends, units, and multipage positioning errors around CharXiv, OCRBench v2, CC-OCR or LongDocURL.
  4. Long video time reasoning: Fixed frame budget, compare the effect of uniform frame drawing, event retrieval, subtitle assistance and timestamp prompt on Video-MME tasks.
  5. Side effects of multimodal preference optimization: Compare the changes in the model after SFT, DPO/MPO/RLHF in safe abstent, visual hallucinations, and detailed explanation.

Summary

The focus of intensive reading of the Multimodal LLM paper is not "who first", but to understand how visual evidence enters the chain of language reasoning. CLIP tells us that natural language supervision can shape visual representations; Flamingo and BLIP-2 tells us that bridge modules can connect freeze models with less training costs; LLaVA Route description instruction data will significantly change model behavior; technical reports such as Qwen, InternVL, Molmo, Llama Vision show that modern VLM has entered high resolution, multi-picture, video, document, preference optimization and agentic interaction stage.

As scientific research training, your goal is to answer after reading the paper: what the model has seen, what has been lost, how to align, what data to learn, what benchmark to be verified, and what failure samples show. To be able to answer these questions can be considered to really understand a Multimodal LLM paper.

References

Search date: 2026-08-09. The following links give priority to papers, project homepage, official blog, model card or conference page; leaderboard, API, model publishing status, and Benchmark's latest score are still recommended to be manually verified before submitting or reproducing the experiment.

  1. Radford et al., “Learning Transferable Visual Models From Natural Language Supervision / CLIP,” OpenAI, 2021. https://openai.com/index/clip/ , paper: https://arxiv.org/abs/2103.00020
  2. Alayrac et al., “Flamingo: a Visual Language Model for Few-Shot Learning,” DeepMind, 2022. https://deepmind.google/blog/tackling-multiple-tasks-with-a-single-visual-language-model/ , paper: https://arxiv.org/abs/2204.14198
  3. Li et al., “BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models,” arXiv, 2023. https://arxiv.org/abs/2301.12597
  4. Liu et al., “Visual Instruction Tuning / LLaVA,” NeurIPS 2023 Oral, project and code. https://github.com/haotian-liu/LLaVA , paper: https://arxiv.org/abs/2304.08485
  5. LLaVA-OneVision documentation, LLaVA-NeXT. https://github.com/LLaVA-VL/LLaVA-NeXT/blob/main/docs/LLaVA_OneVision.md
  6. Bai et al., “Qwen2.5-VL Technical Report,” arXiv, 2025. https://arxiv.org/abs/2502.13923 ,GitHub: https://github.com/QwenLM/Qwen2.5-VL
  7. Qwen Team, “Qwen3-VL Technical Report,” arXiv, 2025; official repo and blog. https://arxiv.org/abs/2511.21631 , https://github.com/QwenLM/Qwen3-VL , https://qwen.ai/blog?id=99f0335c4ad9ff6153e517418d48535ab6d8afef
  8. Zhu et al., “InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models,” arXiv, 2025; project page. https://internvl.github.io/blog/2025-04-11-InternVL-3.0/ , https://github.com/OpenGVLab/InternVL
  9. Wang et al., “InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency,” arXiv, 2025. https://huggingface.co/papers/2508.18265
  10. Deitke et al., “Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models,” arXiv, 2024; Ai2 blog/repo. https://arxiv.org/abs/2409.17146 , https://allenai.org/blog/molmo , https://github.com/allenai/molmo
  11. Meta, “Llama 3.2 Vision Model Card,” 2024. https://github.com/meta-llama/llama-models/blob/main/models/llama3_2/MODEL_CARD_VISION.md
  12. OpenAI, “GPT-4o System Card,” 2024. https://openai.com/index/gpt-4o-system-card/
  13. Google, “Gemini 2.5: Our most intelligent AI model,” 2025. https://blog.google/innovation-and-ai/models-and-research/google-deepmind/gemini-model-thinking-updates-march-2025/
  14. Xu et al., “Qwen2.5-Omni Technical Report,” arXiv, 2025. https://arxiv.org/abs/2503.20215
  15. Yue et al., “MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark,” CVPR 2024 Oral. https://arxiv.org/abs/2311.16502
  16. Lu et al., “MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts,” ICLR 2024 Oral. https://mathvista.github.io/
  17. OpenCompass, “MMBench: Is Your Multi-modal Model an All-around Player?” official repo. https://github.com/open-compass/MMBench
  18. Fu et al., “Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis,” arXiv, 2024. https://video-mme.github.io/home_page.html
  19. Chen et al., “MMStar: Are We on the Right Way for Evaluating Large Vision-Language Models?” NeurIPS 2024. https://mmstar-benchmark.github.io/
  20. Wang et al., “CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs,” NeurIPS 2024 Datasets & Benchmarks. https://proceedings.neurips.cc/paper_files/paper/2024/hash/cdf6f8e9fd9aeaf79b6024caec24f15b-Abstract-Datasets_and_Benchmarks_Track.html ,repo: https://github.com/princeton-nlp/CharXiv
  21. Yang et al., “CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy,” ICCV 2025. https://mlanthology.org/iccv/2025/yang2025iccv-ccocr/