Robotics · rather diagnostic
Policies inside generated worlds
Can a learned visual world select a better fixed placement policy than a default policy, measured against held-out native simulator outcomes?
48 cases · 2 seeds · 2 compared configurations · completed 2026-10-16
What this shows. All four policies failed all twelve native resets, so this cohort cannot establish useful policy selection. The automated judge called seven generated clips successful, but an agent audit flagged identity or release ambiguity in all seven. Inspect the matched rollouts; neither the tied reference results nor these unvalidated labels support a policy or world-model ranking.
Compare the native simulator rollout with both generated worlds on the same initial reset and fixed policy. Later actions can differ because each policy receives its own observations. Generated worlds use an approximate no-contact actuator and much slower visual updates than the 5Hz commands.
This clip’s measurements
- AI-labeled success signals (unvalidated)
- 0.0 %
- AI-judged readable outcomes (unvalidated)
- 0.0 %
- World generation time
- 7.4 seconds
One registered input and seed. Study averages and uncertainty are shown below.
This clip’s measurements
- AI-labeled success signals (unvalidated)
- 0.0 %
- AI-judged readable outcomes (unvalidated)
- 0.0 %
- World generation time
- 21.3 seconds
One registered input and seed. Study averages and uncertainty are shown below.
Same registered input and seed. Examples follow the registered case order; the full run data includes every seed and failure. Link to this comparison
Paired differences across all runs
Cosmos 3 Edge − Cosmos 3 Nano. Uses every jointly scored case and seed for these two configurations, across the full study.
| Measurement | Left − right | 95% interval | Paired coverage |
|---|---|---|---|
| AI-labeled success signals (unvalidated)percentage points | -1.0 | -12.5 to +12.5 | 96 matched pairs6 groups · 0 excluded pairs |
| AI-judged readable outcomes (unvalidated)percentage points | +11.5 | -11.5 to +33.3 | 96 matched pairs6 groups · 0 excluded pairs |
| World generation timeseconds | -12.9 | -13.2 to -12.6 | 96 matched pairs6 groups · 0 excluded pairs |
A positive difference means the left model has a higher value; a negative difference means a lower value. Whether that is better depends on the metric. Excluded pairs include inapplicable or unscored measurements. These differences use matched runs and can differ from subtracting the displayed averages. Intervals that cross zero do not establish a clear direction of difference.
| Model configuration | World generation timeseconds · lower is better | Run errors / runs |
|---|---|---|
| Cosmos 3 Edge | 6.96.9–7.0 | 0 / 96 |
| Cosmos 3 Nano | 19.919.5–20.2 | 0 / 96 |
All measurements, method & evidence
| Model configuration | AI-labeled success signals (unvalidated)% · lower is better | AI-judged readable outcomes (unvalidated)% · higher is better | World generation timeseconds · lower is better | Run errors / runs |
|---|---|---|---|---|
| Cosmos 3 Edge | 3.10.0–12.56 groups · 96 scored runs | 70.845.8–92.76 groups · 96 scored runs | 6.96.9–7.06 groups · 96 scored runs | 0 / 96 |
| Cosmos 3 Nano | 4.20.0–12.56 groups · 96 scored runs | 59.440.6–81.36 groups · 96 scored runs | 19.919.5–20.26 groups · 96 scored runs | 0 / 96 |
Means weight each seed and scene/group equally. Descriptive 95% percentile intervals use 2,000 crossed seed and group bootstrap draws, seed 202610320; cases remain nested within groups and paired differences use the same draws. An unscored seed/group cell withholds the balanced aggregate instead of silently reweighting it. Scored observations, excluded pairs and failures remain visible. Small cohorts and evaluator bias limit these intervals.
When every resample gives the same result, the interval cannot describe unseen successes or failures. No observed variation does not establish certainty.
Development agreement was 12/12 native, 2/2 frozen and 7/8 agent-audited generated clips. A post-scoring agent audit flagged identity or release ambiguity in all seven primary S labels. This targeted, non-blind review is not independent validation; original scores remain available but cannot establish a calibrated false-success rate.
AI-labeled success signals (unvalidated). Original registered fraction of all native-failure cases assigned S by the AI judge. Every S label has an agent-audit ambiguity flag. This is a retained evaluator output, not a validated world-model false-success rate. U contributes no S signal.
AI-judged readable outcomes (unvalidated). Original fraction assigned S or F by the AI judge, with U separate. Primary S-label audit found identity/release ambiguity; this is evaluator coverage, not adjudicated readability or correctness.