For graduate students, early-career researchers and research engineers; search and experiment date: 2026-09-12.

Abstract
Once images and captions have been encoded as vectors, comparing them can look like a single matrix multiplication. The tricky part is deciding which pairs should count as correct. If “red square” describes several images, why reward only the one in the same row? We reconstruct the dual-encoder alignment procedure from the official implementation, train a minimal model on CPU, and check the positives and denominators in both retrieval directions. Even incorrect pairings produced very low training loss. Perfect top-one text-to-image hit rate also failed to establish that every relevant image had been retrieved.
Contents
- Reproducing Value: Determining the Proposition to Validate
- Core ideas and formulas
- A guide to the official implementation
- A minimal experiment and its results
- Evaluation protocol: related collections are more important than the row number
- Troubleshooting and limits of the evidence
- Questions for further research
- Summary and References
Reproducing Value: Determining the Proposition to Validate
CLIP is a contrast-based language image pre-training: two encoders process pictures and text respectively, and map the two inputs to a comparable representation space. Radford, Kim and others published their original papers in ICML 2021 to study whether large-scale graphic supervision can support migration. The "paper report" here is only used to define the research background, and the natural image performance in the paper is not regarded as a cost result. Paper Publishing Page
The proposition of this verification is narrower: whether the alignment loss can learn a given pairing relationship, and whether it can distinguish "relationship pair" and "row number pair". The value of the small model is that each input, gradient and ranking can be saved, and the interference of download weights, graphics cards and distributed training is eliminated. The cost is also clear: simple color shape tasks cannot prove natural language understanding, open vocabulary migration or real image robustness.
Core ideas and formulas

The two channels in the figure remain independent and are compared at the similarity matrix. Training and retrieval are two ways to use a matrix; green diagonals indicate a single paired batch, which does not mean that all data has only one positive sample.
The picture after projection is expressed as , the text is expressed as , then
Each vector has a weight; a batch Generate two for the input matrix and one Fraction matrix. The normalization removes the influence of the length, and the dot product becomes the similarity of the cosine; It is a logarithmic scale, which is often written as a logarithm of the reciprocal of temperature. The scale changes the sharpness of the distribution.
In the first The picture is only with the first When the text is paired, the symmetric target can be written as
The first item is searched line by line, and the second item is looking for pictures one by one, accounting for half of each; other items in the same batch are used as negative samples. This corresponds to the core calculation of Figure 3 of the original paper. body text
My interpretation is that when two captions are identical, rewarding only one row turns another semantically valid match into a competitor. Before asking whether a larger batch helps, check whether the annotations justify that training objective. A high similarity score tells us the model favors a candidate in this representation space. It cannot independently verify every count, relationship, and attribute in the caption.
A guide to the official implementation
from OpenAI CLIP repository Enter, first input and output according to the interface, and then go deep into the network. The repository provides a model and preprocessing loading interface, and the successful running inference example cannot be equated to the original pre-training. The following is an actual static check, and the official model has not been implemented.
The first stop is clip.py: load Returns the model and preprocessing at the same time. Images are specified for zooming, center cropping, RGB conversion and channel normalization; tokenize The default length is 77, and the default error of long input is reported, and truncation can be explicitly selected. If you change a set of crops and use the weight of the same name, it will change the experimental conditions.
The second stop is model.py: tracking encode_image、encode_text Go to forward. After the text takes the representation of the position of the end marker, the feature returned by the two independent coding interfaces also needs to be normalized, and the dot product with the exponential scale and its transposition are calculated in the complete forward direction. Missing this step when manually calling the encoder is a positionable implementation deviation.
The third stop to see the pinned version of the community OpenCLIP v2.32.0 ClipLoss. It is not the original author's complete training script, but can be checked for bidirectional cross entropy, as well as label offset and feature collection in multi-card local loss. Changing the global negative sample to a local batch will change the training problem, and you can't just compare whether the code can run.
The OpenAI source code is read the same day main, the submission is not fixed; the network request to obtain the submission number failed. Subsequent reproduction experiments should add hash, weight check value, dependency and data version. Whether the dynamic source code has changed, and the official complete operation result is "waiting for manual verification".
A minimal experiment and its results
Code is minimal_clip.py, environment and complete agreement see Dependency Description. Actually run with Python 3.12.14, NumPy 2.3.5 and CPU double precision, independent of PyTorch. The experiment directly generates an RGB array of sixteen by sixteen: red, green and blue are combined with squares, crosses, and rings into nine categories, and then randomly change the center, brightness and background noise; no pre-trained image features are used.
Each seed generates 108 training pictures and forty-five test pictures, twelve and five in each category. The text is a nine-two-word description, represented by a six-dimensional bag of words, that is, each dimension indicates whether a color or shape word appears. The picture is flattened into seven hundred and sixty-eight dimensions; each side is projected to sixteen dimensions with an unbiased linear mapping, with a total of 12,384 parameters.
Nine pictures in each batch, one for each category, avoid repeated descriptions of training batches. Fixed 400 steps, Adam learning rate 0.01, scale ten, do not learn the temperature of the original model. The three seeds are 71, 72, and 73, and the data disturbance, initialization and sampling are changed at the same time; the number of training steps is pre-fixed, and checkpoints are selected without passing the test score. The two groups of normal pairing and loop mismatching share initialization and image batches.
Execute in the code directory:
python -m pip install -r requirements.txt
python -m py_compile minimal_clip.py audit_results.py
python minimal_clip.py
python audit_results.py
The installation command is for readers to use; this reuse has an existing operating environment. The script prints training pixels, text, embedding and fractional shapes, and saves data, weights, loss per step, and complete test matrix.
The following table is all measured in this synthesis task, the unit is a percentage, and the mean and sample standard deviation of the three seeds are reported; it is not a Confidence interval, and it is not a reproduction score of the original paper.
| experimental conditions | Graphic search Hit@1 | Text search map Hit@1 |
|---|---|---|
| Untrained initialization | 2.96 ± 1.28 | 11.11 ± 11.11 |
| Normal graphic pairing | 88.89 ± 6.67 | 100.00 ± 0.00 |
| Fixed cycle mismatch | 1.48 ± 2.57 | 0.00 ± 0.00 |
The last loss of the normal group of seed 71 is 0.003447, and the mismatch group is 0.003498, which is almost the same low. This is a deliberate counter-example: the optimizer can seriously learn the wrong data association. Therefore, "loss drop" only means that the designated training target is fitted and cannot be used as an alternative to independent semantic tag evaluation. actual log
Evaluation protocol: related collections are more important than the row number
The picture search text uses 45 test pictures to query nine descriptions, each picture has one related candidate; the text search map uses nine descriptions to query forty-five pictures, and each description has five related pictures. Search the complete test gallery on both sides, and pretend to be a complete search without a small batch ranking. The matching is based on the combination of color and shape, and the candidates are in ascending order according to the original index time, and the protocol is fixed before calculation.
here is the number of queries, is the former (k) item, are all related items. The former asks "have you found one", and the latter asks "how many relevant items have been retrieved". This article is explicitly named separately, avoiding only writing r@k with unknown meaning. When the first place in the text search map is correct, Hit@1 is one, and the recall of the relevant items is only one-fifth; the two values of the same experiment are correct.
Another deterministic check establishes an ideal similarity matrix of nine groups, three fully equivalent members of each group. The first hit of the semantics is one, and if the line number is misused for the only answer, it is one-third. At scale ten, the diagonal cross entropy is 1.098975; add the probability of all positive samples in the same group and then take the negative log number to be 0.000363, and the difference is just ln(3). This is the experimental evidence that the multi-positive sample training program is better than the original target.
At the same time, record the mean MRR of the last related result ranking, and the two indicators of the previous one, three and five items. Expanding the gallery will change the difficulty of the task, and increasing the duplicate item will change the probability denominator; only the scores that are consistent with the query, the candidate set and the relevance label are suitable for direct comparison.
Troubleshooting and limits of the evidence
In this independent audit, the training function was not imported, but the ranking of 3645 cosine values was rebuilt by multiplying the weighted scalar and adding 486 queries. The maximum difference from the save matrix is 1.22E-15; the maximum error of the numerical gradient of the thirty-two small inspection parameters is 1.29E-9. The training test pixel hash has no intersection, and the grammar, smoke test and result audit are all passed. audit record
The normal group has a total of 15 image searches in the first place, all of which are the same color and different shapes. The author infers that the color information is easier to retain, and the outline after translation is more difficult to distinguish by linear mapping; this is not a causal conclusion. To verify the reason, the location should be fixed, the color should be removed, or the convolutional encoder should be replaced, and the same test protocol should be retained.
The check sequence can start with the input: first see whether the crop is removed, the description is cut off, then check the normalization and label replacement, and finally check the negative sample and the gallery. Multiplying all numbers to the same positive scale will not change the sorting when the embedding and candidates are fixed, so the inference temperature of the single change cannot be repaired by these sorting errors; changing the temperature during training will change the gradient, which is another experiment.
The data hash does not repeat only the exact same pixels are excluded. This experiment is trained and tested to share nine combinations, and cannot claim that there is no concept generalization. original CLIP Model card It is also pointed out that counting, fine-grained recognition and language scope are limitations; the success of the English synthesis description cannot be extrapolated to be valid for Chinese retrieval. Official weights, real data sets, multi-card training, video memory and delay have not been tested, and wait for manual verification.
Questions for further research
The first question is how the positive sample is defined. The single pairing, batch deduplication, and multi-positive sample targets can be compared in a multi-text collection of artificially marked, and at the same time check whether the key attributes are omitted for the missing description. If the "red object" is roughly equivalent to the "red square", it may also cover up the wrong shape recognition.
The second problem is combinatorial generalization. Some color and shape combinations are reserved in advance to ensure that each individual attribute will still appear during training, and then report separately from the current new sample test. This tests whether the model is recombined attributes, rather than identifying only the nine categories that are familiar.
The third problem is whether the difficult negative sample is valid. Add candidates of the same color or the same shape and different colors, first count the wrong type, and then compare the method gain and variance. The above are all research plans, and there is no numerical benefit to be implemented or announced this time.
Summary
The first step in reproducing the CLIP is to connect coding, normalization, pairing, and retrieval evaluation into a checkable chain. This minimum experiment verified the objective function and ranking protocol, and also exposed the relationship between low losses, mispairing and multi-positive samples. The next step of the reader should recalculate the saved matrix, and then replace the encoder or real data; every time the scope of the experiment is expanded, it will re-explain what the evidence can support.
References
The following links are actually opened on 2026-09-12, and the title, author or maintenance subject and version have been checked.
- Twelve authors such as Alec Radford, Jong Wook Kim, etc., 2021, Learning Transferable Visual Models From Natural Language Supervision,ICML / PMLR 139:8748–8763; Publishing text PDF, for two-way goals and research backgrounds.
- OpenAI, CLIP official repository, the day main; clip.py and model.py, used for preprocessing, encoding and similarity paths. The submission number is not fixed, and it will be checked manually.
- ML Foundations / OpenCLIP contributors, ClipLoss implementation, fixed v2.32.0 label, community implementation; for loss and multi-card tag reading.
- OpenAI, CLIP Model Card, the day of the main, the model date is marked in January 2021, including the subsequent release records; it is used for the open boundary between ability and data.