AMD says failure-data fine-tuning and PPO fixes improved reproducibility in robot world-model RL, while action-fork errors remained.

AMD’s technical article says reinforcement learning inside a robot world model becomes unreliable when the model mislabels failures as successes, and that both data repair and training-recipe fixes were needed to make gains reproduce across seeds. The report, published on 20 October 2026, focuses on the RoboTwin adjust_bottle task and argues that outcome fidelity matters more in a world model used as an RL environment than in one used only as an evaluator.

According to AMD, the study used BWM, an action-conditioned video world model, as the environment and π0.5 as the policy. The authors measured how often the model judged failed simulator rollouts as successful, then retrained the world model on failure rollouts and reran PPO inside the repaired model. They say the base BWM drew 82.5% of simulator failures as successes, a gap that pixel-level metrics barely reflected.

The article says failure-data fine-tuning cut that fake-success rate to 26.3% while preserving success recall at 0.964. The repair used 677 rollout trajectories from π0.5 and several simulator-PPO checkpoints, including 165 failures, and changed the data only, not the architecture or loss. AMD says the result suggests that broad visual similarity is not enough when the model is used to supply reward during policy optimization.

AMD reports that a corrected PPO recipe inside the fine-tuned world model fixed three defects: truncation bootstrapping at the episode limit, the point where the critic attaches, and the value learning rate.

Source link