
- Date: 2026-11-02
- For graduate students, early-career researchers and research engineers
- Data deadline: 2026-11-02; focus on the search for first-hand research in the past 12 months, and do not claim to fully cover. This article is a deep reading of the paper, and the model experiment is not run.
Abstract: When the object in the figure is said, which layer of evidence is missing?
The model replied "You can drink tea on the table, because there are teapots and cups", and also point out the location of the cup. Such an explanation is easier to check than a "yes", but still leaves a question: the cup really affects the answer, or does the model find a suitable statement after the model decides the answer? The cover is made of tea sets as conceptual illustrations, not a sample of the paper, nor is it an experimental stimulation of strict matching.
This article compares three studies: checking visual descriptions, deleting objects to observe and predicting changes, and checking the consistency of description and judgment with paired images. The current evidence supports the individual assessment of visual basis; it does not yet support the use of smooth interpretation, correct positioning or high-answering score as a proof of transparent internal reasoning. The three papers did not form a simple approval and opposition camp, the main difference is which relationship "faithfulness" refers to. [1][2][3]
Take the "trust" first
The multimodal thinking chain is the intermediate inference step of the model for image and problem output; this article mainly discusses the visible text explanation, rather than all internal calculations. Visual landing, referring to a sentence description that can correspond to an object, attribute or relationship in the figure. Behavioral loyalty further asks: explain the emphasis on evidence, whether it is consistent with the response of the model to predict the input change.
There are at least three distinguishable explanations here. First, the model uses the evidence of the cup and reports it accurately in the reasons. Second, the model does see the cup, but mainly answers based on other clues, the reason is only correct but incomplete scene description. Third, the model is wrongly perceived or hinted at hints, and a reason that sounds reasonable is generated. These three situations may happen to answer correctly.
For ease of understanding, you can do a thought experiment that has not yet been performed: keep the teapot, remove the cup, and ask "is there a drinking water container that can be used directly". If the model sticks to the original answer, it is necessary to check whether it also counts the teapot as a container, and cannot immediately announce that it is not looking at the picture. The meaning of intervention must be decided by the title and label rules together; the "key object" in the reader's intuition is not necessarily the necessary evidence in the definition of the task.
Tier 1: Are the visual statements in the interpretation true?
Uppaal et al. "Journey Before Destination" divides the inference chain into perception steps and inference steps, only judges whether the image is in line with the perceptual step, and uses the model referee to count the infaith perception rate. The author also compares the reliability of the referee with manual annotation; this does not mean that the referee has the quality of artificial gold labeling on any task. 1, §3 and Table 1
This study uses the inference model derived from the same base model in the perception task, and regenerates with the help of external detector positioning problem steps. It reports that the visual description can be improved, but Table 4 shows that after switching to a weaker detector, the description index may be improved, and the accuracy decline may also be accompanied by a decline. Additional detection and rewriting also adds to inference costs. Therefore, the results cannot be expressed as "models are more reliable by self-reflection, and the perceived sentences can not be equivalent to these sentences that cause the answer. 1, §5–7, Table 4 and limitations
The understanding of this article is: this layer is suitable for answering "whether there is a wrong picture", not "whether or not to answer the whole reason". If an explanation only says "there is an object in the picture", it may rarely go wrong, but it will hardly help. Evaluation should also care about correlation, omission and information volume; otherwise, deleting specific details may also improve surface indicators. This is an analysis of the measurement objectives, not a verified cheating experiment in the paper.
The second layer: the object that affects the prediction, did the explanation mention it?
Menzat et al. delete the image pairs of a single object, ask the original image and the editing picture respectively, measure the influence of the intervention with the change of the probability of the candidate answer, and then judge whether the explanation is mentioned to the deleted object. Its vCCT examines the correlation between the magnitude of the impact and the mention; the vCT only counts the "reversal of the answer without mentioning" as a failure, so it is easy to get a high score when it is not flipped. 2,§3
Across two task types and eight models, vCCT was higher when the model answered before explaining than when it explained first. The measure concerns visible explanations elicited by particular prompts, and probabilities are normalized only over a fixed candidate set. It cannot characterize every native reasoning trace. The authors controlled for editing artifacts by reconstructing images without deleting an object, though both editing and the detection of object mentions can still fail. The order changes this behavioral measure. That finding does not establish that an explanation written afterward recovers the internal computation. 2, §3.3–4 and Table 1
This result may seem counterintuitive, but it actually exposes two different questions: "explaining whether the input changes can predict which changes will affect the answer" and "whether the answer depends on the explanation that has been written" is not the same thing. The first answer and then the explanation cannot be used as a pre-step of the answer that has been completed at the same time, but the relevant input may still be described. Conversely, writing an explanation first does not guarantee that every sentence is necessary to calculate.
For researchers, the value of deleting objects is to actively change the evidence. However, the large change may also come from the unnatural picture, the occlusion relationship change or the failure of the candidate answer. If the only object that can answer the question is removed, but the forced model is selected from the original option, the response may be the response when there is no suitable answer. At this time, you should first judge whether the question can still be answered, and then explain the probability movement.

Figure: Proposed measurement process. The reconstructed diagram of the original image and the retained content and the deleted object diagram are handed over to the same model, and then the answer changes and evidence are mentioned. The arrows only indicate the information processing sequence; the dotted cup marking is removed and should not appear in the real test chart. There is no paper to measure the value in the figure, and it does not indicate the proven causal mechanism.
Level 3: After seeing the difference, can you make it clear?
SPD-Faith Bench, etc., let the model compare image pairs, distinguish whether changes are found, whether they say pairs of objects and changes, and whether they are self-consistent. The contradiction rate is symmetrical about "the same" and "different"; the description of the alignment index is evaluated for interpretation and visual differences. Papers report different dimensions and are not synchronized. 3,§3
The authors also analyze attention and hidden representations to investigate failed explanations and propose an account of how visual information travels through the model. I draw a narrower conclusion: an attention hotspot over the right region cannot by itself establish that an attribute was correctly encoded or used in the eventual answer. The data construction section of v1 describes manual verification in both the past and future tense. I therefore leave open how much was checked and whether errors remain. That uncertainty does not invalidate the reported diagnostics. 3,§3.1、§4
This work prompts an actual reading trap: the model answers "different", and at the same time writes what the difference is, it cannot be regarded as a complete success; Negative sentence comprehension, random sampling, output parsing, and visual recognition may all participate. To locate responsibilities, you need to reserve controls for these links.
Why the three evidences cannot be directly ranked
| measurement | observation entrance | Local judgments that can be supported | can not be proved |
|---|---|---|---|
| Does the perceptual sentence match the image? | Gradually label and referee | Describe whether the error is reduced | Explain whether the answer |
| Explain tracking visual impact? | Object deletion and prediction changes | Whether the reference and intervention impact are related | Full internal causal path |
| Does the description match the judgment? | Image pairs and symmetrical questions | Difference description, logical consistency gap | Failure comes from the unique mechanism |
The three entrances in the table correspond to [1], [2], and [3] respectively. Tasks vary from single-map perception quiz to visual implication, and then to dual-map difference description, the starting point, hint and output structure of the model are different; one ratio is summarized according to the perceptual sentence, one indicator is the correlation between samples, and the other measures the contradiction of paired answers. They are neither common denominator nor waiting for budget competitions.
In particular, it is necessary to distinguish and improve. [1] Introduce detection and rewrite, [2] mainly compares the prompt and measurement protocol, and [3] also discusses the internal characterization and correction methods. If you put them together to declare a certain training method most effective, it will cross the variables actually controlled by the paper. This article only compares the questions that the evidence can answer, and does not splice the list numbers.
A follow-up experiment to distinguish competing explanations
The following is the recommended plan, which is not running this time. First fix a model that can read the candidate scores and a set of issues that explicitly depend on object properties. Prepare the original image, keep the content of the reconstructed diagram, change the key attribute map, and change the irrelevant area map. First confirm that the question still has a valid answer on each picture, save the editing instructions and the actual change area;
Compare the exact same image and question, first explain and then answer, first and then explain. The three modes record the actual output length, image resolution, sampling setting and model call times. When the budget cannot be completely equal, the cost of each model is reported at the same time, and the direct answering comparison of the same total call quota is increased, to avoid the benefits of reading several times of the map to the interpretation form.
Three records that are irreplaceable at the time of interpretation are kept: whether the final answer is correct; whether the statement is supported by the image; whether the key evidence that has been changed is mentioned. It is also necessary to check the direction. For example, "the cup disappears" and "the cup is still there" cannot be counted as success just because there is a "cup". The referee should look at the specific image and sentence, and use independent sampling to annotate the estimation error, and the generative model cannot be announced by itself.
If the explanation can reflect the evidence, the key attribute change should cause a description and prediction change that is consistent with the task, and irrelevant editors should not cause the same range of disturbances. If only the description has changed and the answer is not sensitive to key attributes, it should be checked for alternative evidence or downstream binding failures. If the answer changes but the reason is not updated, it is more supportive to explain the important factors of omission. If the reconstructed map also causes similar changes, the image processing process should be checked first, rather than expanding the conclusion of loyalty.
Further test whether the interpretation participates in the answer, you can fix the image, only replace one attribute in the intermediate interpretation, and then let the model continue to answer; at the same time, add synonymous rewriting and irrelevant sentence replacement. This operation may still make a prefix that the model does not usually encounter, so the impact of the input exception must be reported. Even if the changes in the direction are consistent, only the dependencies under this setting are supported, and the entire reasoning chain cannot be determined according to this.
It is also important to prescribe what results will overturn your own interpretation. If the effect appears only in one editor or one prompt, it is narrowed to the agreement; if the difference disappears after the independent judge is used, the scoring deviation is given priority. Don't just pick the successful example of the answer to flip: keep samples that have no changes, abstension, and undetermined samples to know what the measurement covers.
Summary: A more detailed explanation, to accept a more specific test
The multimodal Think Chain provides a checkable statement, so it has research value. But checking does not mean already believable. Read a paper that claims to be more faithful, and you should first ask whether it checks the facts of the image, the influence of the input, or the effect of the answer, and then sees whether the control is enough to exclude alternative interpretations.
The current conclusion of this article is that it is more reliable to report these three layers of evidence separately than to put the general label of "can see pictures" or "just make reasons" to the model. The real question is whether these measurements can be improved together under the same task, similar budget and independent judges, and maintained in new visual conditions.
References
- Rheeya Uppaal、Phu Mon Htut、Min Bai、Nikolaos Pappas、Zheng Qi、Sandesh Swamy,2026, Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking. EACL 2026 formal paper; read this article Published paper.
- Bayar Menzat、Maximilian Süss、Ruizhi Wang、Benno Steinegger、Thomas Lukasiewicz、Oana-Maria Camburu,2026, Counterfactual Tests for Measuring Chain-of-Thought Faithfulness in Visual Language Models. Findings of EMNLP 2026 official page. The official PDF has not been obtained due to the size limit of access, the method and result basis arXiv V1 full text, do not claim to check the final publication verbatim.
- Weijiang Lv、Yaoxuan Feng、Xiaobo Xia、Jiayu Wang、Yan Jing、Wenchao Chen、Bo Chen,2026, SPD-Faith Bench: Diagnosing and Improving Faithfulness in Chain-of-Thought for Multimodal Large Language Models. Findings of ACL 2026 official page. The official PDF has not been obtained due to the size limit of access, and the basis for the experiment discussion arXiv V1 full text.
Apps by the author
The following is the author's own product promotion. In addition to writing a technical blog, I am also doing some small tools that make everyday life more organized. Welcome to the two apps I independently developed:
- Daily Expense Journal: Record daily accounts, arrange budget and savings goals. View on the App Store.
- One Small Action: Select small actions according to the current time and energy, and record the feelings and discoveries after completion. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.