Off-policy RL with online inference tuning.

RoboEval is a set of eight tasks for two Franka arms in the MuJoCo simulator. Every episode starts with the objects in new random positions and orientations. The main point of the benchmark is not only whether the robot gets the job done, but how it behaves on the way: is it safe, is it efficient, do the two arms work well together. I trained one policy for all eight tasks.

Success is only part of the score

Each episode first gets an outcome: 70% of it is for succeeding and 30% for how many stages of the task you got through. Then the challenge looks at how you moved: safety (collisions and jerky motion), efficiency (time and path length) and how well the two arms worked together. All of these are compared with the human demonstrations, and moving like a typical human already gives full marks.

The behavior part is multiplied by the outcome, so a failure hurts twice: you lose the outcome points, and the behavior points shrink with them. Once a policy moves about as well as a human, the only way to gain more is to fail less and to hit every stage.

• I started from π_0.5, a proven vision-language-action model, with the same architecture I used in my previous competitions.
• My first approach was the most general one, carried over from LeHome: only the organisers' demos plus the policy's own rollouts, with RL on the challenge score. It reached 0.85 in a couple of days and was heading for 0.9.
• It was clear that the demos themselves would stop it well short of 1.0, so I added a lot of better scripted demos. After about a week of training that took it to 0.98.
• I spent the last two days on test-time optimisation, picking the final checkpoint and lots of testing. That gave me another 0.01 and brought the score to 0.99.
• The benchmark is basically solved: the top three teams all score around 0.99, and the few failures left could probably be fixed with more task-by-task tuning.

One policy handles all eight. It is only told which task it is on, with no text instruction. Pick a task to watch a real episode and to see how it scores next to the human demos.

The submitted policy on a held-out seed, in real time. Human demos: every demo replayed and scored like a policy (0.875 on average). My policy: 500 held-out episodes per task.

I started from the public π_0.5 weights: a vision-language model with a flow-matching action expert. It is the same base I used for LeHome and for our BEHAVIOR 2025 solution. The orange parts are what I added.

• Value query: looks at the cameras and the task, and predicts how likely the episode is to succeed, which stages it will reach and how much behavior score is still to come. It runs during data collection too, so the critic is the same model that collected the data.
• Q head: sits on the action side and predicts how much a candidate action chunk would change those values.
• Advantage input: an extra token that tells the policy whether to act like a “good” episode or an average one. That is what makes classifier-free guidance possible.

Who sees what

All of this is one transformer, so the attention mask matters. The value query only sees the images and the task, so it never sees the advantage it is used to compute. The advantage token is the switch that guidance flips.

This is the heart of the project. Training, data collection and inference tuning all run at the same time. The trainer never stops, and the collector always runs the newest policy.

Collection scales to more machines out of the box. I tried two collectors but mostly used one.

The reward is the challenge score itself. Every step gets credit for how much it moves each part of the score, weighted the same way the score weights it, plus a small bonus for making progress.

The baseline is the policy's own value estimate, used only as much as it can be trusted. The nice part is that this reward knows what an improvement is worth right now: making a path shorter than the human one earns nothing, while on StackTwoBlocks every centimetre saved still counts.

The advantage is then used twice: through the sampler, so good frames show up more often in training (AWR), and as an input token inspired by RECAP, so that guidance can push the actions towards “good” at test time.

A policy call here is much more than one sample. I used a lot of test-time logic:

• Best of N: up to 6 action chunks are sampled in one batch, and the Q head picks the one it thinks will do best.
• Guidance: each candidate is denoised with and without the advantage input, and classifier-free guidance pushes it towards “good”.
• Chunk inpainting: only the first few actions of a chunk are executed. The next few are carried over to the start of the next chunk, so it continues where the last one was heading (in the spirit of real-time chunking).
• Stall boost: if the predicted success drops or stops growing, the next three calls sample with more noise and use all the candidates.

Tuning it with Thompson sampling

That is about 20 settings per task, and I didn't set any of them by hand. Every collected episode runs with its own settings, and Thompson sampling keeps moving each task towards the settings that score best. The same episodes are also the RL training data, so the policy and its settings improve together.

Speed is my favourite example. A policy trained on demos moves more slowly than the demos themselves, so early on the tuner played chunks 1.5–3.9× faster. RL then taught the policy the right pace, and the speed-up went back to about 1×. Here's where each task ended up, all found automatically:

Final inference parameters. Best of the first k of 6 candidates, ranked by predicted change of success or of the whole score.

The human demos show strategies that are far from the best ones for this score. My loop can tune speed and smoothness, but it can't invent a completely new strategy: RL can only reward a stage that some rollout actually reaches. Here are the three clearest cases.

Top: human demo from the RoboEval dataset (Wang et al., 2025). Middle: my script. Bottom: the learned policy on a held-out seed.

Writing my own demos

So I wrote a scripted policy for each task. The scripts read object positions straight from the simulator, so they can't be submitted, but they make great training data: they get the full reward in more than 99.5% of episodes, and I generated around 200k of them.

Each script picks its own speeds, offsets and grasp points every episode and reacts to what happens, so the policy sees many slightly different ways of doing the task. Every cycle adds a fresh batch, with some of the robot state hidden so the policy has to rely on the cameras.

The policy played more than 240k episodes of its own. The failed ones are the most useful, because they show exactly where it goes wrong.

So every failed or partly failed episode is replayed up to a branch point (the failure, 2 s before it, or the last moment it looked good) and continued from there, first by the policy itself and then by a script. Only continuations that reach the full reward are kept, about 100 per collect.

Two recoveries from training, in real time: the policy's failed rollout, then the continuation from a branch point. Left: Lift pot. Right: Stack two blocks.

I worked on this for about nine days. Here's the whole run in one curve: first the generic loop, then a switch to lots of scripted data.

RDS of every collect over training steps: 2,000 held-out episodes each, every one with its own inference settings, so the curve runs below the tuned score. Mean per 2k steps. At step 42k training moved from one H200 to eight B300s with a bigger batch. Hover or tap for values.

Once the curve levelled off, I stopped training and spent the last two days on a big search over weight averages and inference settings, starting from a few checkpoints around step 50k.

Extra evaluation runs are cheap, so every option was tested on new random seeds. That mattered: with dozens of candidates and differences around 0.005, the winner on one set of seeds is often just the luckiest one. An option only went in if it won again on seeds that played no part in choosing it. All that for about +0.01. It matters on a leaderboard, but in a real deployment it probably wouldn't be worth the effort.

The submitted policy scores 0.990 with 99.6% success over 4,000 held-out episodes.

500 held-out episodes per task, each task at its final parameters. Full reward: success with every stage reached.

Coordination is 1.000 everywhere, and every task succeeds more than 98% of the time. What's left is mostly behavior: the StackTwoBlocks path is longer than its reference, which comes from one-arm demos, and on PackBox some contacts happen at reset, before the policy even moves.

Final results

The final results of the challenge will be published on November 12, 2026. Before that, the organisers re-run the top entries on unpublished seeds.

What still fails

Failures are very rare: 16 in 1,589 extra episodes on the five tasks where it fails most. Here are three of them.

Failures of the submitted policy on fresh held-out seeds. Episodes longer than 10 s play at 2×.

What it took

• My off-policy RL loop works well. This is the second competition I've used it in, after LeHome, and both times the results were very high. The core loop only needs rollouts and a score, so it carries over to new tasks easily, though getting the most out of it here still took some task-specific work. Squeezing out the last points with the loop alone just isn't very data-efficient.
• The last stretch was about data. Scripted demos work when you can trust the simulator and the task is easy to script, which isn't always true. They had a cost too: trained on much cleaner data, the policy lost some of the regrasps and retries it had picked up from its own messy rollouts. Still, for tasks like these it worked very well: it's what took the score from the mid-0.8s to 0.98.
• Near 1.0, every improvement costs more compute. Getting to around 0.9 was fast, but after that every extra bit took more rollouts, evaluation and training than the one before.

License. The code and the checkpoint are released under Apache-2.0. The checkpoint is a modified π_0.5 (openpi).

Credits. The benchmark, the simulator and the human demonstrations are from RoboEval (Wang et al., 2025); the demonstration clips and frames on this page come from the helen9975/roboeval_ee_delta_20hz_scaled dataset on the Hugging Face Hub. Everything else is rendered with the challenge environment. I used Claude Code to run the experiments.