AI Frontier Observation: Characters, Breakthroughs and Evidence | Interpretation Date: November 7, 2026 | Information Deadline: November 7th at 20:33 (Beijing time)
The robot places the last block. The tower stands, but the letter is sideways. From the final frame alone, it’s hard to tell whether the model misread the letter’s orientation or got it right and the gripper set the block down crooked. On November 6, the Allen Institute for AI (Ai2)introduced ToyBlocks, bringing both possibilities onto the same simulated tabletop. Keep the model’s decision, swap out the component that executes it, and see where the block lands. That design tells us more about why a small tabletop makes a worthwhile research platform than a count of completed towers alone.

Schematic diagram of original concept: The target pose and the actual action can be checked separately. Non-experimental photo or result map.
Who can see the correct answer?
Suppose the goal is to spell the specified characters from bottom to top with three blocks. For a robot that only sees the camera screen, there are two explanations for not finding the letters: there is really not on the table, or that one is sticking to the desktop. The latter needs to be turned first to decide whether to grab or not. Even if all the literals are facing forward, whether the words are upright, which one is placed first, and how to release the grippers may make mistakes respectively. A tower looks like it's over, but it's still one direction away from the completion of the mission.
The simulator has an advantage that the real camera does not have: it knows the exact position and pose of each block. ToyBlocks uses this information to generate a demonstration, and the planner arranges the skills of flipping, grabbing and putting into an executable sequence of actions; the trained model is then decided according to the sensor observation. The difference between the two is very important: the teacher knows which side the building block is facing, and it does not mean that the student can recognize it from the pixel. Likewise, Item page The first batch of 85,336 long-range demonstrations listed are automatically generated simulation experiences, not so many real robotic arm tests, and there are not so many different tasks. For researchers, it is more useful to decide which literals are covered in the next batch of demonstrations than simply increase the video duration.
The same tower, who will take it?
In a system tested by the team, the Vision-Language Model Molmo 2 looks at the image selection skills and targets the target; the action model ManiFlow receives the command and turns it into the movement of the robotic arm. To judge whether the action model has learned to place it, you can first let the symbol planner point to the building block, and then observe the performance of the gripper. Conversely, keep the Molmo 2 command and use the reference skill to generate the demonstration to see whether the step it selected can move the task forward. Evaluation Documents These two substitutions are given, and also allow restarting from the initial state of the record.
The system will also note the progress of the stage. A three-story tower, even if the last one is crooked, the upper and lower relationship between the first two pieces is still established. Keep the conditions that have been met one by one, and you can distinguish the rounds that you choose in the wrong place, and the round that is only the last place to be placed.
This gives us evidence beyond a replay of one failure. If both executors faithfully pick up the wrong block, it makes more sense to inspect the image the higher-level model received and the location it pointed to than to keep adjusting the gripper. But changing the executor can also change where the block lands and hence what the higher-level model sees in the next frame. The gap in success rates cannot be read directly as the fraction of failures caused by one component alone. Find the first step where the two trajectories diverge: were the instructions still the same, and did the object move as expected?
The fifteenth success of the real machine, what is the denominator?
The real machine comparison of the team Using the Franka FR3 robotic arm and Robotiq jaws, each scene has four to ten blocks that you have seen during training. The three-character spelling selects the easiest start condition: the target character is visible and upright. Molmo 2 is used in conjunction with ManiFlow, 111 of the 200 rounds are completed in the simulation, and 15 of the 30 of the 30 are completed on the real machine. A success here means that the entire task meets the requirements; the operator stops due to unsafe actions, which is also considered a failure. It answers whether this system can complete a series of actions, and cannot be used to estimate the success rate of finding hidden literals.
The ratios on both sides are 55.5% and 50% respectively. In 30 real machine tests, one more success will make the ratio move about 3.3 percentage points. Based on these two points alone, it is difficult to distinguish between environmental differences and sampling fluctuations, let alone determine the equivalent of the two environments. There is also a condition that is easy to miss: Although the strategy is only trained with analog data, the team first calibrated the parameters of the simulated robot with the real machine trajectory. "No real machine demonstration training strategy" has a clear technical meaning; it does not mean that it will get the same result in the face of a new robot that has never been measured.
What records are worth keeping in the next set of trials?
A useful follow-up test, you can keep the original building blocks, commands and robotic arms, first change one thing: turn some target characters to the back. For each starting layout, in addition to recording whether the entire tower is spelled correctly, also write down when the model first saw the target character, whether the block was selected, and whether the orientation was correct after grabbing and placing it. In this way, even if the final success rate is reduced, it is possible to distinguish whether the model has spent the action budget and is still looking for words, or has it been found but missed the last place to place it. Open repository It is supported to send back the failed status to the demonstration generation process; if the training is supplemented accordingly, a batch of layouts that have never been used to make up the lessons should be left to verify the benefits.
This is still a test to be done. The announcement, project page, code and document of Ai2 are from the same team. There is no independent team to redo this group of real machine comparisons in this search, and this article does not run experiments. What ToyBlocks has given is a way to leave a comparative record of failure. The next more weighty evidence will be whether others can find the same weak link according to these records, and in the desktop layout that has not been seen before, just change this step to make more entire towers fight.
References
- Ai2 Research Announcement: Design, Scrabbling and Real Machine Results of ToyBlocks
- ToyBlocks Project Page: Scale and Composition of Public Demonstration
- ToyBlocks Evaluation Document: Replace Planning or Execution Module, Round Judgment and Playback
- Ai2 official code repository: data generation, failure status and strategy evaluation
Apps by the author
I also build small tools for everyday life. Here are two of my apps:
- Daily Expense Journal: Track your income and spending, plan a budget and set savings goals. View on the App Store.
- One Small Action: Pick a small action that fits the time and energy you have today, then record how it felt and what you noticed. View on the App Store.
Find more articles and projects on my website: zffjzz.top. I’d love to hear how the apps work for you and what I could improve.