ROWBench: Do Video Models Render What the Program Specifies?

0:00Welcome to Daily Papercast. Today, we are discussing a paper from the Hugging Face Daily Paper List of October 2, 2026, with 29 upvotes. The paper is titled, ProWBench. Do video models render what the program specifies? It's authored by Zhang Hui Huang, Guishu Lin, and others from a Liolab, with Zhisheng Huang as the corresponding author. Let's dive into the introduction. Ashley, can you set the stage for us? What background do they provide? Sure, Evan. The authors start by highlighting recent advances in video world models, which have enabled increasingly realistic, open-ended interactive environments. However, a significant challenge remains, maintaining persistent world states and enforcing user-defined rules. This becomes complex when entities, attributes, and interaction outcomes are represented implicitly through visual histories or latent context.

1:01That sounds like a fundamental problem. How do they propose to address it? The paper discusses programmable world models and code world models. Essentially, these models separate the evolution of world states from visual generation. By using coding agents and executable programs, states and interaction rules are maintained explicitly, while video models generate observations conditioned on structured representations of the world. So, just because a program executes correctly, does it ensure that the visuals will be faithful to that execution? Exactly. Even with explicit state control, a generated video can appear plausible while violating rules or failing to depict specified interactions. This raises the central evaluation question of the paper. Can video models faithfully render the scene structure, rules, and interactions of a programmable world? Interesting. What about existing benchmarks? How do they fit into all of this? Good question.

2:01Existing benchmarks focus on aspects like video quality, dynamics, instruction following, and interactive control. However, the paper points out that these benchmarks rarely test fidelity to fine-grained program-specified events. Without replayable records of entity states and timestamped events, it's difficult to judge a generated video against its program execution. So, what do the authors propose with pre-WBench? The authors introduce ProWBench, a benchmark specifically designed to evaluate the visual realization of programmable worlds. ProWBench combines a programmatically constructed data set with complementary metrics that assess logic adherence and interaction realization. The benchmark logs entity states and timestamped events as replayable world records and renders them into synchronized views and proxy representations. And what does this allow researchers to do? This setup allows researchers to check generated videos against the observable consequences of program execution.

3:04The data set includes 170 dynamic scene episodes with three synchronized representations from shared state records, allowing for controlled comparisons. The authors also evaluate entity control and long horizon memory, introducing two VLM-based metrics, logic render alignment and interaction success rate. So, with logic render alignment, they can check if actions in the timeline are visually depicted and interaction success rate assesses whether recorded events appear as they should. Precisely, and they have an extensible programmatic framework to separate scene construction, behavior control, state recording and rendering. This allows researchers to expand layouts, actions and representations while generating aligned observations from shared state records. Fascinating. This could make a significant impact on how we evaluate video models. Definitely. This concludes the introduction of the paper, setting the stage for the methods and experiments to come.

4:05All right, Ashley. Let's delve deeper into the methods used in this paper. How do they construct ProWBench? And what makes their approach unique? Sure thing, Evan. The construction of ProWBench is centered around a programmable world generation pipeline, which includes world synthesis, control compilation, and a final process for exporting videos and structured annotations. Their pipeline is designed to produce replayable world trajectories and synchronized conditioning inputs for video models. Let's break that down a bit. What's involved in world synthesis? World synthesis is the first step in their pipeline. This involves constructing dynamic worlds from detailed configurations that specify environments, entity layouts, actors, task controllers, seeds, and execution duration. Procedural routines generate terrains, buildings, roads, interiors and interaction objects, ensuring that everything is set up according to a semantic structure. Entities are provided with persistent identifiers, semantic categories

5:08and task relevant attributes for the episode. So they build a highly detailed and controlled world from scratch. What roles do the task controllers have? Task controllers are essential for coordinating poses, object trajectories and interactions. They support behaviors such as interaction phases, animal behaviors, flight and swimming dynamics, and driving. For instance, if a task involves carrying an object, the interaction controller will manage approaching, grasping, lifting, transporting and releasing the object, updating hand targets and object ownership accordingly. That's detailed. What happens after the world synthesis? How do they get these worlds ready for evaluation? After world synthesis, they move to control compilation. This step involves compiling the recorded world states and camera trajectories into synchronized visual conditioning inputs for generative video models. Third-person cameras follow task dependent trajectories while first-person viewpoints derive from recorded head positions,

6:09orientations and gaze information. The pipeline then maps these shared world records to conditioning representations like course 3D, semantic proxy, and colored oriented bounding boxes. And what is the final process in tail? In the final process, the pipeline exports scene and motion records, rendered observations, and text and interaction labels for validation. Scene and motion records enable replayability, including scene geometry, entity identities and semantic categories, spatial states, articulated poses, bounding volumes, entity and camera trajectories, and camera parameters. Rendered observations consist of synchronized proxy videos across viewpoints and representations with supplementary depth interpretations. How do they ensure the quality and consistency of these outputs? The validation involves task and timeline checks, geometric checks for consistency, replay and alignment checks, and visual and media quality checks. They assess whether the recorded events and states match the observed videos

7:11ensuring fidelity to the program dynamics. Unsuccessful attempts are logged separately from accepted episodes, preserving provenance and revision history. That's robust. They seem to have thought of every detail. What metrics do they use to evaluate the generated videos? They propose several complementary metrics. For entity control, they measure spatial alignment using bounding box intersection over union, IOU and center error. For camera control, metrics like rotation error, translation error, and camera mean consistency, CAMMC, are used to assess how closely generated camera paths follow the reference trajectories. And how about visual and temporal quality? For visual quality, they use image inequality and aesthetic quality from established predictors, war-bearer to measure temporal consistency and motion smoothness to assess the continuity of motion across frames. Clipscore evaluates how well the video content aligns with the textual description.

8:13What stands out to me are their unique metrics for logic and state alignment. How do they measure this? That's where logic render alignment and interaction success rate come in. Logic render alignment checks if the actions prescribed in the timeline are visibly rendered within the corresponding time window. Interaction success rate evaluates if registered engine events are visually realized, assessing both the action and expected end state using frames sampled around the event window. These metrics seem critical for assessing adherence to the programmed world states. How do they handle long horizon memory and multi-view consistency? For long horizon memory, they look at reappearance IOU, measuring spatial recovery of entities that leave and re-enter the view and state persistence of returning entities, judging if they maintain the prescribed locomotion state. Multi-view consistency employs metrics for appearance compliance and consistency across synchronized views, ensuring each participant looks the same across different viewpoints.

9:16It sounds like ProW Bench's design ensures a deep and thorough evaluation. Any final points on how they organize this benchmark? ProW Bench organizes evaluation cases into two main tracks, a main track for control generation of short clips, and a challenge track for long horizon and multi-view evaluations. Each track is divided into settings that use their own evaluation sets, ensuring a comprehensive assessment of the model's capabilities under various conditions. Thank you for breaking that down, Ashley. This concludes the methods section and really sets the stage for understanding their experimental results next. All right, Ashley, let's dive into the experiments and results. How do they evaluate the models in ProW Bench? Great question, Evan. The experiments are organized into a main track and a challenge track. Each track includes specific settings designed to test different aspects of video model performance. For the main track, they evaluate the ability to generate five second clips under two settings,

10:20verified FF, where models are given an initial reference frame, an unverified FF, where they aren't. And how do these settings differ in terms of evaluation? In verified FF, the models start with a provided RGB first frame that anchors the appearance and layout. While an unverified FF, models must generate the appearance purely from text descriptions and proxy videos. They use various metrics to assess performance, including bounding box intersection over union for entity control, camera mean consistency for camera paths, and specific visual and temporal quality measures. Can you give us an overview of their findings in the main track? Certainly. In the verified FF setting, proxy conditioned models like Lin Real Omni, Mini Max H3, and CWM, perform significantly better in B-Box IOU, ranging between 0.454 and 0.518. Camera conditioned world models like Echo WM and Sonna WM, however,

11:21perform at lower B-Box IOU levels, between 0.303 and 0.351. For logic render alignment, models like Lin Real Omni and C-Dance 2.5 also lead, showing high realization of the prescribed timelines. Interesting. What about their performance in the unverified FF setting? In the unverified FF setting, the trends are much the same. Models like Lin Real Omni and Mini Max H3 achieve higher B-Box IOU scores compared to camera conditioned models and perform well in following the prescribed entity trajectories. This reflects that even without an initial frame, proxy conditioned models can maintain the alignment between the generated video and predefined actions. How do they approach the more challenging evaluations? The challenge track evaluates more complex scenarios, including long horizon settings, where clips last 30 seconds and multi-view settings where each scene is observed from multiple cameras.

12:22These tests examine entity control, memory over long periods, and consistency across different viewpoints. What did they find with these longer and multi-view clips? In the long horizon setting, the models struggle significantly more, with B-Box IOU scores dropping to between 0.175 and 0.280. The findings highlight challenges in maintaining state fidelity across time. For memory and appearance IOU, scores also indicate room for improvement, with entities' reappearances scoring between just 0.053 and 0.131. In multi-view settings, identity consistency across views becomes crucial, and models like Minimax H3 and Lin-Rial Omni perform better in ensuring that the same participant appears consistent across all views. Those seem like tough conditions for any model. Were there any standout aspects? Yes, despite the overall complexity, models like Minimax H3 showed strong consistency in both short and multi-view clips,

13:24managing comparably high appearance compliance and cross-view consistency scores. How about their results on the domain-specific cases, like the gunfight set? Good reference, Evan. For the gunfight set, they included PWM, a model specifically trained on gunfight scenes, to compare general-purpose models against a specialized one. PWM notably exceeded others in interaction success rate, achieving an ISR of 0.850 compared to 0.650 or lower for other models. This shows that specialized training still holds value for domain-specific tasks. They seem to have a comprehensive benchmark. Any last thoughts on their findings? An important takeaway is how different proxy representations influence adherence to programmed events. The colored-oriented bounding box proxy generally resulted in higher interaction success because it provides a clear semantic link between entities and actions, unlike more abstract proxies.

14:26That's insightful. It makes clear how critical proper representation is in these tests. And it points to the need for balanced and well-designed benchmarks, like ProWBench, to push the capability of video-generation models. This wraps up the experiment section. Next, we'll be exploring related work and conclusions from the paper to understand where ProWBench fits in the broader research landscape. Let's move on to the related work section, Ashley. How does ProWBench compare with other benchmarks and methodologies in video world models? Evan, the related work section of the paper is quite thorough and touches on various benchmarks, methodologies, and models that have contributed to the field. The authors begin by discussing video world models and programmable world states, focusing on how different approaches have evolved. Could you elaborate on these video world models? Certainly. Video world models typically predict future observations from visual context and actions.

15:30For instance, Genie learns controllable environments from unlabeled videos, while Game Engine enables real-time action-conditioned game generation. Models like Game Factory, Matrix Game, and Lingbot World Advanced Generalization, Interaction, and Long Horizon Consistency. However, these models in first states from visual history and memory, which has its limitations. And how do programmable world models differ? Programmable world models externalize state evolution. Code world model uses a coding agent to maintain executable states and interaction logic, while programmable world model feeds persistent structured states to a generative renderer. This separation allows for more reliable state control and interaction realization, which is precisely what ProWBench evaluates. That makes sense. How about structured representations for visual generation? Structured conditions offer direct spatial control through elements like depth, trajectories,

16:30bounding boxes, masks, and camera motion. Various models incorporate these methods, including geometry-aware techniques using 3D boxes and bird's-eye view layouts. Generative rendering uses explicit geometry or renderer-derived signals, with approaches like course to reel and video from 3D utilizing course 3D scenes, while diffusion renderer and a lyre renderer use denser rendering cues to span control from sparse object details to dense geometric representations. It's evident that structured representations play a significant role. How do the benchmarks vary in evaluating world models? That's right. Several benchmarks assess video quality, temporal consistency, motion, and semantic alignment. Notably, VBench and Eval Crafter measure these aspects, while WorldSimBench, WorldModelBench, and WorldScore extend evaluation to physical plausibility and controllability. Mind, WorldMark, OmniWorldBench, WorldArrina, and IWorldBench target interactive behavior, memory, and long horizon consistency.

17:32These benchmarks tackle a variety of elements, but often couple errors in action understanding, state prediction, memory, dynamics, and rendering. So how does ProWBench stand out among these benchmarks? ProWBench differentiates itself by focusing on the fidelity of visual realization, based on recorded states, rules, and interactions. It uses engine-recorded frame-level states to isolate rendering fidelity, and allows comparisons across synchronized perspectives and proxy representations under the same trajectory. The logic-renderer alignment measures whether specified rules are visibly realized, while interaction success rate evaluates whether recorded events appear at the corresponding times. This specificity in evaluating adherence to the programmed world states is a key distinction. That's quite focused. How do they address multi-view and long horizon evaluations? They tackle these evaluations by organizing them into specific tracks. ProWBench includes settings for the controlled generation of short clips and more challenging long horizon and multi-view settings.

18:39Long horizon settings test if a model can maintain world state overextended periods, which is crucial for game worlds and simulations. Multi-view settings assess whether synchronized views maintain consistency in entity appearance and behavior, ensuring participants agree on world dynamics across different observations. This must impact how researchers and developers approach video model evaluations. Any final comparisons or benchmarks they discuss? The authors emphasize the need for benchmarks that handle executable states and interactions over time and across views. They mention models like world play, matrix game 3.0, and a LiA world, which improve persistence through spatial context retrieval or reconstruction. These models contribute to improved entity control and long-term consistency, but Proto-Black Bench's unique benchmarking on visual realization puts it in a distinctive position. It's clear that ProWBench occupies a unique place in the landscape of video model benchmarks.

19:42That concludes the related work section. Alright Ashley, let's summarize the key contributions and takeaways of this paper. What are the main highlights? Sure Evan, the paper makes several significant contributions. Firstly, it introduces Proto-WBench, which is a benchmark comprising 170 dynamic scene episodes and 600 synchronized proxy videos. These cover diverse environments and interactions, enabling grounded comparisons and reliable evaluations of video model performance. And it's not just about the data set, they propose metrics for comprehensive evaluation, right? Exactly, two novel metrics stand out, logic render alignment and interaction success rate. Logic render alignment assesses whether the prescribed actions in the timeline are depicted within their corresponding segments, while interaction success rate checks if recorded events are visually realized as per the engine logs. They also cover long horizon memory and multi-view consistency, which are critical for real world applications.

20:48Yes, their benchmark includes tests for long horizon consistency to see if models can maintain world states over extended periods, and multi-view evaluation to ensure synchronized views across different perspectives maintain entity appearance and behavior. This thorough approach provides a granular view of model performance, emphasizing fidelity to program states and interactions. By isolating rendering fidelity from state dynamics, Proto-WBench offers a robust foundation for future research. The use of synchronized proxy videos and detailed logging of entity states and interactions makes it possible to evaluate adherence to the prescribed world dynamics very precisely. Well, that wraps up today's episode. Thanks for tuning in to our discussion about Proto-WBench and its contributions to video model evaluation. We hope you found this episode insightful. Make sure to join us next time as we delve into more cutting-edge research from the world of AI and machine learning.

21:53Thank you for listening, and until next time, keep exploring and stay curious.