Physical Self-Play

Today, we are announcing a breakthrough post-training result for physical AI via self-play. Our result shows that a strong base model like S1 can learn to complete extremely dexterous and dynamic tasks, like soccer, by competing against itself in a simulation.

Last month, we introduced S1, our flagship robotics foundation model, that learns to perform tasks from in-context demonstrations similar to how language models learn from their prompts, which enables an extraordinary breadth of capability.

However, because S1-class models are pre-trained on human data, they are capped at human capability. We believe physical self-play will enable robots to far exceed human capability.

The Self-Play Era

Self-play ushered in the age of artificial intelligence long before the invention of language models.

AlphaGo beat world champion Lee Sedol in 2016, famously playing the “inhuman” move 37. Just three days after beginning self-play, AlphaGo Zero defeated the original AlphaGo 100 to 0. Self-play was later extended to multi-player games like StarCraft II with AlphaStar and Dota 2 with OpenAI Five, which defeated the world champions in 2019.

By discovering novel strategies that were not present in their training data, self-play allowed AI to complete tasks of staggering complexity at a super-human level.

The concept of self-play for recursive improvement subsequently lost momentum, as reinforcement learning from verifiable rewards emerged as the simpler, more immediately viable approach.

With these exciting new results, we hope to revive the field’s interest in self-play and transform it from a relic of digital AI to a fire-starter for physical AGI.

A Physical Revival

We gave the model a single objective: score. The policy then learned to score by playing against recent versions of itself. As it improved, its opponents improved alongside it — turning every gain in capability into an increasingly difficult opponent.

In its first few simulated months inside NVIDIA’s Isaac Sim, it could barely walk. By the time it was college-aged, it could stand back up after falling down.

In time it taught itself new capabilities: dribbling past the defender, shielding the ball, tackling its opponent. Rather than hand-crafting rewards for these behaviors, they emerged simply because they helped the model score.

After 140 years of simulated play, we transferred the policy into a robot and challenged it to a match:

We see the successful transfer from simulation to humanoid, along with the sheer complexity of the task, as a bright green flag for self-play on the road to physical AGI.

Why soccer

Robot soccer cups already exist, but performance is still far below human level. There is also growing interest in other types of robot sports. Soccer especially is a great testbed for robotics and self-play. The game requires both physical and strategic ability.

However, the approach is general, and we are extending this approach to several day-to-day robotics tasks beyond sports.

At Scale

This simulation ran for over a century with a trivial goal. What happens when we run it for orders of magnitude longer, on a virtual construction site, factory, or home?

We have just scratched the surface of physical self-play, and are already quickly making progress applying it to other domains. In our next release we’ll explore the social behaviors that emerge as we scale to larger teams, from collaborative manipulation to city-scale navigation.