Bringing AI for robots that act in the real world, often called Physical AI, into practical use has drawn attention to fine-tuning VLA (Vision-Language-Action) models, which produce robot actions from images and language, with reinforcement learning. However, a numerical “success” is not always a real success in the robot’s behavior.
An independent implementation of RL Token, one such method, led us to reconsider how robot learning measures success. Numbers that initially suggested improvement turned out to measure how often an orange reached the vicinity of a plate. We combined human video review, an AI agent implementing changes, and visual audits by AI to redesign evaluation and rewards.
We used an SO-101 robot in the LeIsaac simulator, with the task of moving three oranges onto a plate. The experiment independently implemented RL Token using π0.5, a model that produces robot actions from images and language.
The large model stays frozen. Its internal representation is compressed into a small, 256-dimensional representation called an RL token. Only lightweight components that choose actions and estimate their value—the actor and critic—are trained. The aim was to test whether this approach could improve behavior without updating the entire large model.
The initial metric rose from 45% to 50% in one setting. A later experiment with rewards for intermediate behavior showed a rise from 30% to 70%. Both figures, however, measured whether at least one orange was near the plate under the old position-based criterion. They were not rates of correct placement. Strict task success, requiring all three oranges and a stopped pose, remained 0% for every policy.
A person reviewing nine videos discovered the discrepancy. The old criterion used a box extending ±10 cm from the plate center in both horizontal directions, with a height window that included the tabletop. It never checked whether the orange was actually on the plate. Pushing an orange along the table toward the plate could therefore count as success.
This was reward hacking: behavior could earn a reward without achieving the intended outcome. We needed to revise the success criterion that guided learning, as well as examine the learned behavior.
We therefore rebuilt everything from the success criterion to the reward. We first measured the plate to define the region that counts as placement, then combined a circular region relative to the plate, height, stillness, and release from the gripper into a stricter success criterion. Next, we implemented reward v3, which awards a placement reward once per orange, and added conditions to the rewards that guide intermediate behavior. We also tested six adversarial trajectories, namely pushing, hovering, grasp-and-release, dropping, holding without placing, and crossing the rim, to check that false placements earn no placement reward. Finally, placement events are audited by a vision-language model and passed to a person for judgment and approval. Each issue found during human review led the implementation agent to revise the criteria or rewards and rerun the tests.
The implementation agent handled measurement, criterion and reward design, implementation, tests, and preparation of review materials. At checkpoints, a person judged materials with the decision labels hidden and approved whether to proceed.
The vision-language model evaluator, or VLM judge, visually audited placement using multiple views and measurements such as height and gripper opening. Its role was triage: identifying suspicious or disputed cases for a person to review. After calibration against human judgments, the VLM judge agreed with people 98.6% of the time. Even so, the human retained final authority.
The original AI frame checks asked only whether the robot was grasping an orange, missing the placement failure. When two models were instead asked whether placement occurred inside the plate rim, both judged that all nine old videos contained zero placements inside the plate. Both also rejected all three videos that the old criterion had falsely marked as successful, agreeing with human review.
Plate tipping and movement were initially noticed only by a person. When explicitly asked about plate movement, tilt, and contact, both models detected all seven cases of strong disturbance. A concrete result was that including the relevant failure in the audit question made previously missed problems detectable.
Limits remained. For small movements near a threshold, both models could agree with each other while disagreeing with the measurement-based criterion. In examples where an increase in height was misread as transport, disagreement between the models provided a warning. Agreement alone does not establish correctness. Humans also struggled to follow the sequence of motion in still-image strips, so human review materials were shifted toward video.
As of the September 19, 2026 presentation, the first run with reward v3 comprised 50 episodes and recorded six placement events. We verified that the reward calculation was connected correctly and mathematically consistent, but had not achieved improvement through learning. A second run of 400 episodes was awaiting human approval of success criterion v3.4.
The interim result was a concrete process for identifying false positives in the old metric and revising and testing success criteria, rewards, and audit questions. Further work will use human video review and route uncertain or conflicting AI judgments to people, while testing whether the revised reward improves learning.
Combining human video review with AI agents such as a VLM judge is a form of human-in-the-loop work. It surfaces failures that numerical metrics miss and makes revising evaluation and rewards more efficient. We believe this approach will also inform evaluation design for future proof-of-concept projects and real-world deployment.
Yuta Oriike presented this interim report on September 19, 2026, at the 15th meeting (iHR15) of the Intelligent Home Robotics (iHR) technical committee of the Robotics Society of Japan, held at Soka University on September 19–20. The presentation was also introduced in an article by the Nakazawa Laboratory at Kanazawa Institute of Technology, where Oriike is a visiting researcher. Related introductory article (note)
