See how models behave.

Means weight each seed and scene/group equally. Descriptive 95% percentile intervals use 2,000 crossed seed and group bootstrap draws, seed 202610360; cases remain nested within groups and paired differences use the same draws. An unscored seed/group cell withholds the balanced aggregate instead of silently reweighting it. Scored observations, excluded pairs and failures remain visible. Small cohorts and evaluator bias limit these intervals. Different conditioning-access groups are shown separately and have no cross-group paired comparisons.

When every resample gives the same result, the interval cannot describe unseen successes or failures. No observed variation does not establish certainty.

Landmark calibration agreed on 24/24 known controls, 24/24 short generated and 11/12 long generated agent labels. Camera evaluator qualified on 16 known warps and negative controls. Generated first-seed returns were visually audited; independent human validation remains pending. Semantic-category consistency can hide changed object geometry.

AI landmark retention. Fraction of the three registered landmarks labeled P at fixed returns, averaged within session. Broad visible semantic retention estimated by an AI judge; not validated object identity or complete game state.

AI landmark contradiction. Fraction labeled F: visibly absent, replaced or structurally contradicted. An incorrect camera return can also change visibility; this does not isolate a memory mechanism.

AI landmark uncertainty. Fraction labeled U because occlusion, framing or ambiguous identity prevents judgment. Unknown is separate from both retained and contradicted.

Return judgments completed. Returns with successful evaluator responses, including the one explicitly disclosed same-image transport retry. Any unresolved return withholds the complete session landmark average.

Judge first-attempt coverage. Successful original evaluator requests divided by all requested returns, before the disclosed transport recovery. Execution failures remain separate from generated scene quality.

Turn pairs tracked. Fraction of half-second commanded-motion pairs with sufficient SIFT/RANSAC inliers for a measured homography. Coverage must accompany the conditional error.

Rotation pairs qualified. Fraction of commanded-motion pairs whose measured homography also passes the registered pure-rotation residual check.

Commanded warp discrepancy. Session median pixel displacement discrepancy between tracked homography and commanded yaw, divided by image diagonal. Uses assumed 90-degree intrinsics; conditional on tracked pairs, not true 3D pose error.

Qualified rotation error. Session median rotation discrepancy after homography calibration and pure-rotation qualification. Assumed intrinsics and non-rigid foreground limit interpretation.

Qualified yaw direction. Share of rotation-qualified commanded-motion pairs whose estimated yaw sign follows the prescribed yaw. Reports only the qualified subset.

Idle warp discrepancy. Session median commanded-warp discrepancy during no-yaw pairs with sufficient matches; foreground animation can contribute.

First decoded worker chunk. Native generation start/action callback to first decoded chunk. MG3 first chunk contains 57 frames; SANA 24 generated frames. Excludes model loading and browser/network. SANA first session includes compilation.

Steady worker throughput. Decoded frames after the first chunk divided by remaining native stream time. MG3 includes between-chunk encoding overhead; SANA uses its native stream summary. Neither is browser throughput.

Peak allocated GPU memory. Maximum native PyTorch allocated GPU memory for this session. Excludes driver/context and reserved-but-unused memory.

Session process time. Recorded per-session process wall time. SANA fresh-process startup differs from MG3 shared loaded-process execution; not an equal-startup serving comparison.

Late / early worker latency. Mean last-quarter over first-quarter chunk generation latency after excluding the first chunk. SANA lacks per-chunk telemetry and stays undefined; no inferred value.

• Four deliberately varied familiar WBench images; no claim of held-out images or exhaustive commercial coverage.
• Turn-away/return tests visible spatial persistence, not object manipulation, inventory changes or general game-state memory.
• SANA native lattice trims a small tail; evaluation timestamps are shared. Different access and prompts stay explicit.
• No human or physical-environment validation. Automatic landmark judgments can be wrong.
• SANA receives future camera poses and a caption; no paired ranking against next-action-only MG3.
• Worker timing excludes browser/network; separately retained three-chunk browser pilot is not a long-session timing claim.
• One-frame SANA alignment corrected from pinned native source before primary scoring; raw outputs preserved.
• Primary camera-error scores are conditional on tracked pairs; inspect both tracking and rotation coverage.
• Shared H100 class does not control host CPU or storage contention. Browser timing is a separate short pilot; no full-session end-to-end latency claim.
• Minute and stress cost allocations partition the same study rentals; do not add full rental totals again.
• Rental schedules differ: MG3 Base uses one session per rental, while Distilled and SANA share one rental across sessions. Deadline cutoffs are operator execution failures, not model-quality outcomes. Partial outputs remain in the archives; no replacement generation is hidden.

Single H100 80GB per native worker, batch one. MG3 50/3 steps; SANA native streaming path with separate component terms. · $56.24 · Allocated rental estimates plus judge response usage and separately identified no-response reservations. Full ledger and exclusions: Open execution and cost records ↗ (opens in a new tab)

Four familiar MIT-licensed WBench inputs, attributed to WBench authors / Meituan LongCat and Fudan University; fixed new Rather schedules and seeds. Not an official WBench score.

Matrix-Game 3 Distilled. Apache-2.0 released weights and pinned native code. Commercially usable open-weight configuration; not a production-reliability certification. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Matrix-Game 3 Base. Apache-2.0 released weights and pinned native code. Commercially usable open-weight configuration; not a production-reliability certification. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

SANA-WM Streaming. SANA Apache-2.0; LTX-2 and Gemma components retain separate terms. Rather qualifies under the reviewed component terms. Other deployers must check their own eligibility; this is not an unconditional enterprise license. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Means weight each seed and scene/group equally. Descriptive 95% percentile intervals use 2,000 crossed seed and group bootstrap draws, seed 202610320; cases remain nested within groups and paired differences use the same draws. An unscored seed/group cell withholds the balanced aggregate instead of silently reweighting it. Scored observations, excluded pairs and failures remain visible. Small cohorts and evaluator bias limit these intervals.

When every resample gives the same result, the interval cannot describe unseen successes or failures. No observed variation does not establish certainty.

Development agreement was 12/12 native, 2/2 frozen and 7/8 agent-audited generated clips. A post-scoring agent audit flagged identity or release ambiguity in all seven primary S labels. This targeted, non-blind review is not independent validation; original scores remain available but cannot establish a calibrated false-success rate.

AI-labeled success signals (unvalidated). Original registered fraction of all native-failure cases assigned S by the AI judge. Every S label has an agent-audit ambiguity flag. This is a retained evaluator output, not a validated world-model false-success rate. U contributes no S signal.

AI-judged readable outcomes (unvalidated). Original fraction assigned S or F by the AI judge, with U separate. Primary S-label audit found identity/release ambiguity; this is evaluator coverage, not adjudicated readability or correctness.

World generation time. Sum of five native world calls for an 80-transition episode. Includes decoding; excludes policy, simulator reset, model loading, browser and transfer.

• One task, one embodiment, small Rather-trained recurrent policies; not a broad pretrained VLA claim.
• Learned visual worlds plus a no-contact actuator approximation are not full contact-dynamics simulators.
• 80-step Rather extension, not the official Simpler score.
• Independent humans and physical robots unavailable; validation packages prepared.
• Visual feedback is 0.3125Hz, commands 5Hz. This is a slow visual feedback interface, not a 5Hz observation world or real-time robotics deployment.
• Automated judgments estimate visible outcomes, not physical truth. Generated calibration has few clear failures; no independent human review.
• Differences reflect the combined visual world, slow feedback and no-contact actuator approximation.
• Degenerate resamples do not bound unobserved failures or successes.
• All seven positive judgments warrant independent adjudication for identity or release/support ambiguity. The original registered scores are preserved, not relabeled. Neither the automatic labels nor this targeted agent review establish a validated outcome-fidelity ranking.
• No calibrated false-success rate, human agreement rate, or model superiority claim from this audit. A zero native success cohort cannot establish useful policy selection regardless of judge quality.

A100 80GB PCIe learned rollouts; A40 reference controls. Fixed training seed, 5Hz commands and 0.3125Hz visual feedback. · $5.92 · Primary evaluation and judging estimate; exclusions in Open execution and cost records ↗ (opens in a new tab)

Rather simulator resets and generated rollouts; pinned SimplerEnv MIT, ManiSkill2_real2sim Apache-2.0 and Cosmos OpenMDW-1.1 sources with attribution in the linked audit.

Cosmos 3 Edge. Released weights use OpenMDW-1.1; the pinned model card permits commercial use. The exact released checkpoint is evaluated without additional fine-tuning. This is a commercially usable open-weight configuration, not a certification of production reliability. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Cosmos 3 Nano. Released weights use OpenMDW-1.1; the pinned model card permits commercial use. The exact released checkpoint is evaluated without additional fine-tuning. This is a commercially usable open-weight configuration, not a certification of production reliability. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Means weight each seed and scene/group equally. Descriptive 95% percentile intervals use 2,000 crossed seed and group bootstrap draws, seed 202610290; cases remain nested within groups and paired differences use the same draws. An unscored seed/group cell withholds the balanced aggregate instead of silently reweighting it. Scored observations, excluded pairs and failures remain visible. Small cohorts and evaluator bias limit these intervals.

When every resample gives the same result, the interval cannot describe unseen successes or failures. No observed variation does not establish certainty.

All native step outcomes, fixed final checkpoints, complete registered grids, source hashes and every decoded video are checked. Crossed seed/group estimates match the separately retained analysis. Simulator truth does not establish physical-robot performance.

Held-out placement success. Native simulator placement predicate true throughout the final five control steps. Equal-weight spatial groups and training seeds; every registered held-out attempt included.

• One simulated carrot-placement task and a small recurrent policy architecture.
• Held out from Rather training/development, not an audit of world-model pretraining data.
• Generated arms replace RGB while retaining reference action and qpos labels.
• No physical robot or independent human validation.
• Only three training seeds and six spatial groups. Degenerate resampling intervals do not bound unobserved successes or failures.
• All evaluation videos are simulator rollouts of the trained policies; generated videos are training inputs only.

H100 image generation and matched policy training; A40 native simulator evaluation. Exact rental records linked. · $4.34 · Main study rental estimate; exclusions and recovery accounting in Open execution and cost records ↗ (opens in a new tab)

New simulated demonstrations and evaluation rollouts from pinned MIT SimplerEnv and Apache-2.0 ManiSkill2_real2sim, with scene/object attribution. Generated images use the audited Cosmos 3 configurations. Not an official Simpler benchmark score.

Reference images. Rather-trained experimental supporting policies, not production VLA products. The study tests data from commercially usable pinned Cosmos 3 weights; it does not certify deployment reliability or release these policy weights. Exact configuration ↗ (opens in a new tab)

Ordinary image augmentation. Rather-trained experimental supporting policies, not production VLA products. The study tests data from commercially usable pinned Cosmos 3 weights; it does not certify deployment reliability or release these policy weights. Exact configuration ↗ (opens in a new tab)

Cosmos 3 Edge training images. Rather-trained experimental supporting policies, not production VLA products. The study tests data from commercially usable pinned Cosmos 3 weights; it does not certify deployment reliability or release these policy weights. Exact configuration ↗ (opens in a new tab)

Cosmos 3 Nano training images. Rather-trained experimental supporting policies, not production VLA products. The study tests data from commercially usable pinned Cosmos 3 weights; it does not certify deployment reliability or release these policy weights. Exact configuration ↗ (opens in a new tab)

Means weight each corrected analysis group equally; see the disclosed post-publication correction. 95% percentile intervals use 2,000 group bootstrap draws, seed 20261015. Repeated seeds stay within groups. Paired differences use jointly scored attempts; excluded pairs and all failures are reported. These intervals do not measure evaluator bias.

When every resample gives the same result, the interval cannot describe unseen successes or failures. No observed variation does not establish certainty.

The fixed visual rubric agreed with 31/32 simulator references, with one false success among 15 failures and no missed successes among 17 lifts. Four generated connector clips agreed with agent visual inspection. This is not independent human validation; provider alias drift and generated-video bias remain possible.

Outcome agreement. Agreement between the blinded visual evaluator and the simulator sustained-lift predicate, among readable generated outcomes.

Readable outcomes. Fraction of generated clips assigned a determinate success or failure; occlusion, disappearance or uncertainty remains unknown.

False successes. Fraction labeled successful among readable clips whose simulator reference failed. Inapplicable successful references are excluded.

Missed successes. Fraction labeled failed among readable clips whose simulator reference succeeded. Inapplicable failed references are excluded.

Generation time. Sum of three synchronized pipeline calls, including conversion to frames; excludes checkpoint loading, setup, saved-video encoding and API judging. Offline A100 timing, not serving latency.

• Seven distinct simulated layouts from six position-pair groups, one object, one robot and a fixed scripted controller; diagnostic generalization is limited.
• Two configurations of the same NVIDIA model family, not broad provider coverage.
• All 32 references were used to validate the evaluator. One scene was used for four unscored connector clips; full generation uses new seeds. No held-out-development claim.
• No model is given object truth, but the realized robot trajectory is itself supplied; this is not prediction from raw policy commands.
• VLM labels estimate visible outcomes and may share systematic bias. Unknowns and reference calibration errors are reported.
• Latency is warm offline generation on the recorded rented GPU; not network serving latency, real-time control latency or the vendors optimized throughput.
• Frozen image is a deterministic no-dynamics baseline; its repeats do not add independent information.
• Primary error rates use only readable outcomes. Always inspect readable-outcome coverage and all-run evidence.

One rented NVIDIA A100-SXM4-80GB for model generation; RTX 4090 for simulator references. · $5.93 · Estimated GPU rental + evaluator API cost, including failed setup, probes and calibration; excludes storage, egress, tax and local CPU work. Execution/cost ledger: Open execution and cost records ↗ (opens in a new tab)

New simulated footage produced with MIT-licensed SimplerEnv and Apache-2.0 ManiSkill2_real2sim at pinned commits. Simulator code, custom Bridge scene/object assets and authors are attributed in the input manifest. This is not a published SimplerEnv task score.

Cosmos 3 Edge. Released weights use OpenMDW-1.1; the pinned model card permits commercial use. The exact released checkpoint is evaluated without additional fine-tuning. This is a commercially usable open-weight configuration, not a certification of production reliability. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Cosmos 3 Nano. Released weights use OpenMDW-1.1; the pinned model card permits commercial use. The exact released checkpoint is evaluated without additional fine-tuning. This is a commercially usable open-weight configuration, not a certification of production reliability. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Frozen image. Deterministic baseline generated from the same simulated observation. No third-party learned weights. References are generated with MIT SimplerEnv and Apache-2.0 ManiSkill2_real2sim assets/code; attribution is preserved. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Means weight each registered scene/group equally. 95% percentile intervals use 2,000 group bootstrap draws, seed 20261015. Repeated seeds stay within groups. Paired differences use jointly scored attempts; excluded pairs and all failures are reported. These intervals do not measure evaluator bias.

When every resample gives the same result, the interval cannot describe unseen successes or failures. No observed variation does not establish certainty.

Known +/-3-pixel and stationary transformations passed on all 8 images (24/24); textureless frames correctly remain undefined. Full 177-frame synthetic control and frozen-baseline checks passed for all 8 images. A native Matrix-Game 2 probe was also visually audited. This does not validate semantic memory or metric bias on every generated artifact.

Action response. Fraction of reliably tracked turn-phase frame pairs moving in the requested image-plane direction by more than 0.25 pixels. Undefined below 50% tracking coverage.

Return similarity. Mean grayscale structural similarity between eight pre-turn and post-return frames at 640x352. Static video can score perfectly; inspect action response alongside this metric.

Generation time. Synchronized native pipeline wall time for 177 frames, including native preprocessing and video writing but excluding checkpoint loading and installation. Offline timing, not serving latency.

Motion while idle. Median robust image-plane displacement during no-input phases, at 640x352. Dynamic objects and estimator failure can affect this proxy; undefined below 50% idle-pair coverage.

Frozen during input. Fraction of turn-phase pairs with mean grayscale absolute difference below 0.25 intensity units. Measures frozen output, not 3D movement correctness.

Tracking coverage. Fraction of turn-phase frame pairs yielding at least 20 consistent optical-flow/RANSAC inliers. Low coverage can bias action response and must stay visible.

Offline throughput. 177 decoded frames divided by synchronized native pipeline duration, including writing output videos. Excludes model loading; not an interactive streaming measurement.

Warm generation cost. Native generation duration times the recorded GPU hourly rate. Excludes cold setup, model loading and local scoring; total study cost includes the rental overhead.

• Eight selected scenes do not establish broad generalization.
• Image-plane motion and return-view SSIM are diagnostic proxies, not physical state or semantic-memory ground truth.
• Dynamic objects, low texture and scene changes can make motion estimation undefined; coverage and missing values must be reported.
• Equal and opposite native mouse input requests a return, but does not guarantee the generated camera actually returns.
• The frozen baseline intentionally scores highly on return similarity while failing action response.
• This is a new Rather protocol using licensed WBench images, not a WBench leaderboard score.
• Offline generation timing is not end-to-end interactive service latency.
• Three configurations of one Skywork model family, not broad provider coverage.
• Released native resolutions differ. Image-plane metrics use the same 640x352 canvas; compute results describe each native configuration, not equal-resolution ablations.
• Matrix-Game 3 uses explicitly recorded PyTorch 2.8 / Flash Attention 2 instead of the upstream Torch 2.10 pin, with unchanged pinned model code. This runtime passed native generation probes.
• The museum left-return scene was used in one Matrix-Game 2 connector probe with a different seed. All images were used for metric calibration; no held-out-development claim.
• The 11.06-second horizon checks a short camera-return interaction, not long-horizon memory. No semantic, 3D-state, real-time-serving or physical-consistency claim is made.
• Frozen-image output is reused across seeds and does not add independent baseline repetitions.

One NVIDIA H100 80GB HBM3 (SXM) per worker, same hardware class for all three model configurations; CPU scoring is separate. · $50.83 · Estimated GPU rentals including setup, failed probes and cancelled compilation; excludes storage, egress, tax and local CPU scoring. Ledger: Open execution and cost records ↗ (opens in a new tab)

Eight preselected first-person navigation images from the MIT-licensed WBench dataset, pinned to its recorded revision and attributed to WBench authors / Meituan LongCat and Fudan University. New Rather controls and measurements; not an official WBench result.

Matrix-Game 3 Base. Apache-2.0 released weights and pinned native code. Commercially usable open-weight configuration; not a production-reliability certification. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Matrix-Game 3 Distilled. Apache-2.0 released weights and pinned native code. Commercially usable open-weight configuration; not a production-reliability certification. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Matrix-Game 2 Universal. MIT released weights and pinned native code. Commercially usable open-weight configuration; not a production-reliability certification. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)

Frozen image. Deterministic repeat of MIT-licensed WBench input images, with authors and pinned source revision attributed. No additional model weights. Exact configuration ↗ (opens in a new tab) · License ↗ (opens in a new tab)