
The large model research controversy: AI goes into everyday life
The cover is a fictional campus open day scene.
On Saturday morning, someone in the parent group asked which door to enter on the open day. You post the notice to the AI, ask "Which door the parents who arrive at the school from nine o'clock enter the school, and it answers the east gate. The child took the mobile phone, opened another conversation, and posted the same notice, asking "Which door should the parent arrive at the school at nine o'clock", but Ximen appeared on the screen. The explanations on both sides are very polite, but it is you who walks one more stop along the street. This example is fictitious. If I really encounter this matter, I will first flip through the notification to find the entrance corresponding to "nine o'clock", and then put the two questions and two conclusions together.
Here are a few changes that are easy to mix together. If the child is actually asking "9:30", and the notice happens to be changed to Ximen after 9:11, it is justified to change his mouth. If the child asks "Is it not to go to the west gate", the "yes" in the answer may still point to the east gate. What is really worth asking is that, The same person, the same notice, the same time of arrival at the school, the two answers have been restored to the door to go, why did they point to two places?
How to see that one sentence brings in one more condition
Try to give others a pair of questions first. Change "Where to enter the school at nine o'clock from parents" to "Which entrance for parents to enter the school at nine o'clock", it is easy to say which line you need to check in two sentences. Change the last sentence to "Which door is still open at nine o'clock", and things are subtle. The door is open, maybe it is for the staff to move the table. You can know it's on, but still don't know if parents can get in from here. The two interrogative sentences share the easiest words to notice, but there is a difference in the evidence that can be used to judge the answer.
Even those who specialize in rewriting have to be dealt with one by one. Alhetelah and Ahmad used the model to draw up two hundred questions, and each question wrote five more questions. An author checks whether the meaning is reserved one by one, and adjusts or regenerates when it encounters ambiguity or offset. In the paper report, about 7% of the automatic rewriting requires a small manual adjustment. This ratio is about the process of making the questions, not the ratio of the wrong answer to the model. Also because the verification is completed by one person, we don't know which questions will be drawn by changing a reader. Paper method and table 1
There is an example worth stopping in the table. The original question is whether it is in favor of students to receive higher education for free, a rewrite and clearly mention the burden of public funds, and designate for all citizens. Imagine a reader who supports the private scholarship to cover all students, while at the same time there is another set of views on who is eligible for public funding. She can choose different answers in two questions, and she can also explain the reasons. This is my reading of the question. The paper does not interview such a reader, and there is no evidence that the actual answer change is caused by these words. The test that readers can do is try to make up a specific situation, so that the two sentences are established and not established respectively. If it can really make up, it is worth re-discussing whether they are considered the same question. Original table one
The researchers asked the five models to face three fixed answers, and asked to only answer yes, no or both. When looking at the results, they first look at which answer is most often seen in each group of questions, and then calculate the proportion of the answers that are the same as it, and finally cross the question to average. If one answer has always supported free education, it can get a high consistency, even if it never states where the funds come from. The average consistency of each model in Figure 1 is close to 90%. This number allows the opposite answer on a certain question, and does not tell us what students, teachers or taxpayers will choose. If you want to know how a real person understands those few sentences, you need to find another study that really asks someone to answer. Method Section 2.4, Figure 1

Brackets put together each pair of questions to be compared. The upper row is only changed, and the lower row is filled in the arrival time. There are no arrows in the picture, and the two rows are schematics of the control design, which does not indicate the measurement result.
Turn "agree" back to the thing it refers to
A set of comparisons designed by Petersen and colleagues is very suitable for thinking about the school gate. In the insurance situation written in English, they ask someone to read a short story to judge whether a rule is applicable to the protagonist. This is the task of reading sentences and judging meanings. It's enough to just look at how they work, you don't need to understand insurance, and you can't use these answers to judge whether they can be compensated. Original method and table 1
For example, there is something broken in the car of Table 1. The policy mentions the audio equipment in the car and the car. The first version of the bad is the laptop computer for work, the one version is the bluetooth speaker with the carry-on, and the other version is the car audio. The reader was hesitating whether the portable speaker counts, but the bad one is the car audio, and there is a reason to re-judgment. The researcher then left the same item in the car, and only changed the last words of the man to make a statement. One please judge "under the scope of the guarantee", and the other kind, please judge "not covered by the guarantee". They also exchanged the order of agreeing and disagreeing with the two options. Original table one
What I want to keep the most is this step. If the person who presses the button chooses "Agree" the first time and "Disagrees" the next time, it can point to "In the range" both times. If you only copy the words on the button, you will change the idea of those who are loyal to the original meaning. Conversely, the two times of agreeing, it looks neat, but the actual judgment is completely overturned. Translate each answer back to "within the scope" or "not within the scope", and then you can see clearly whether the replacement of the item and the problem is the position of the button under the finger. Original Section 4, Figure 1
This experiment has forty-six stories. Each has three contents, two positive and negative questions, and two options sequences, which together are five hundred and fifty-two questions. The 300 people recruited online in the United States did six questions, eliminated the twelve participants who took less than two minutes, and finally analyzed the answers of 288 people. The five sets of model settings each do all the questions. The result of human beings is the judgment scattered on many people. It cannot be read as a researcher and asks everyone to repeat the same question twelve times to observe who is inconsistent. The model is presented slightly differently from the options that people see. original method, table one
In this group of people's answers, when the content of the story changes, the judgment changes accordingly. Statistical tests did not detect the significant impact of positive and negative expressions or option order on the judgment of the population. The three groups of model settings that do not enable reasoning have positive and negative expression effects. There is also a more interesting comparison than changing the brand. After the same GPT-5.2 is changed to the medium inference intensity, the researchers did not detect the positive or negative expression effect, and the model will still change the judgment according to the substantial content of the story. However, in a contextual version of the paper Figure 4, the thirteen people who answered the questions chose to apply. The five sets of model settings are combined with both positive and negative and order, and thirteen times in the 20 model answers are not applicable. The units of the two "Thirteen" are different. The former is several people, and the latter number models are output under different questions. After the answer has become stable, you still have to ask it alone where it is standing steadily. Original Figure 1, Figure 4 and Section 4.5
Sometimes it's the ruler that approves the answer
Back to the door. Suppose the first answer is "Go to the East Gate" and the second time is "Enter from the East Side". When people look at the map, they know that it is the same door, but a registration form that only receives the word "Dongmen" may make the second one wrong. There is also a situation in the opposite direction. Both answers have "East Gate", one of which is "East Gate is only for leaving school". As long as you can find two identical words, even if you answer correctly, you will also erase the difference that can really make people go the wrong way.
Hua and colleagues go back along the verdict. They compare twelve prompt templates on the six sets of questions, and use various methods to judge the answers given by the model. In the case of the scoring procedure that can only recognize the fixed spelling method, the synonymous reply may be lost. After switching to a model that can interpret the meaning of the standard answer, the score fluctuation caused by different templates in some tasks becomes smaller. There is an interesting exception. The program of the math problem will be simplified and the equivalent will be checked, and the fluctuation seen in the judging method is similar. Whether it can identify the different expressions allowed in this question is more close to the problem than simply distinguishing who uses the machine to judge and who uses the rules to judge. Original Section 3
The researchers also used human graders. They sampled 50 questions per set. For each question, graders checked one designated model’s answers under twelve templates against the reference answer. Across six groups, 73% of questions received the same right-or-wrong judgment under all twelve templates. Getting all twelve wrong counts as consistent. For a child checking homework, there’s a practical lesson here: write down what kind of change you care about. Two different mistakes can earn the same red cross, while “one-half” and “0.5” look quite different and can both earn a tick. Here the people graded answers. In the earlier experiment, they supplied them. Those roles provide different points of comparison. Section 4 and Table 3 of the paper
Leave three notes and you can see where the differences are
Here’s a paper exercise you can try. I haven’t run it. Write a notice that says, “On open day, parents arriving before 9:15 should use the east gate. From 9:15 onward, please use the west gate.” Make three slips. On the first, ask, “A parent arrives at nine. Which gate should they use?” On the second, ask, “Which entrance should a parent arriving at nine take?” On the third, change only the arrival time in the second question to 9:30. Attach the same notice to all three. On the back, write the words you relied on, then ask another reader to circle the time and entrance independently. If you disagree about the condition itself, start by finding the word that allows both readings.
If you want to do the model too, put each individual in a new dialogue and keep the original answer. You can ask the first one as it is, and see if the text will be changed when the text is not changed. When recording, leave two columns for each reply, one column says "Which door will go to do as you do", and the other column will copy "The original sentence of the notification based on it". If you encounter a reply to two doors at the same time, see if it explains the time period clearly. If it asks about the time of the school, you can write down the "question" separately. In this way, there is no need to make up two options, and just count a clarification as a wrong one.
The expectations on the paper are very specific. The first two should take the people who arrive at the school at nine o'clock to the same place. The time of the third picture has crossed the boundary line, and it is worth checking if it still insists on walking the same door. If even the first repetition is given different entrances, it is difficult to rewrite the reason based on the difference between the first and the second time. If the rewriting is still going to the east gate but the terms of leaving the school in the afternoon are quoted, the route is temporarily the same, and the reason has been a problem. Show these two columns to the child, he can point out which line is wrong, no need to guess what the model is thinking.
There is also a way to help the teacher who wrote the notice to see the problem. Cover the "from 9:10" in the original sentence, only show the first half of the sentence, and then ask the third note. At this time, it is only 9:30 that it is already late than the time specified in the first half of the sentence. To conclude that Ximen can enter, there is still a second half of the sentence. If the reader is willing to leave a question mark when the evidence is vacant, the question mark will mark the most worthy of the notice. The same approach can also be used to check the extra details in the answer.
The three studies did not observe the Chinese parent group, and did not count how many families went the wrong way. The original sentence can be found on a notice, and some real notices have missed the arrangement of "parents" or "after nine o'clock". In the latter case, I asked the teacher in the group, "Do the parents who arrive at 9:30 also enter through the east gate?" It is better to make up the evidence than to repeatedly rewrite the same question. When the new notification comes, and then change the answer, it is exactly what we hope that both readers and models can do.
References
- Bushra Alhetelah, Irfan Ahmad, 2026. Measuring LLMs' Sensitivity to Paraphrased Opinion Prompts, the official paper of WASSA 2026 workshop. Original
- Dawson Petersen, Abhishek Purushothama, and Nathan Schneider (2026). Sense and Sensitivity: “Reasoning” Models are More Robust, but can Diverge from Human Consensus in a Legal Interpretation Task. Published at CoNLL 2026. Original
- Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, Yao Qin, 2025. Flaw or Artifact? Rethinking Prompt Sensitivity in evaluation LLMs, the official paper of the main meeting of EMNLP 2025. Original
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.