Runway has introduced GWM Worlds 2, a research preview that transforms high-fidelity video and audio generation into real-time interactive simulation using an autoregressive diffusion model. The release features WorldPrompt, a new input format that allows users to specify generated worlds and actions by fixing aspects of the environment and creating timestamped events.

  • WorldPrompt acts as a control layer for characters, cameras, and environments, enabling detailed control over scene subjects similar to computer games.
  • GWM Worlds 2 streams continuous 720p video at 24 frames per second and audio at 48,000 Hz in real-time.
  • The model achieves real-time generation by fine-tuning a foundational audio-video model to the WorldPrompt format, post-training it to generate autoregressively, and applying distillation methods.
  • Co-founder Anastasis Germanidis explained that the process converts bidirectional diffusion into an autoregressive system to generate frames one at a time.

This approach aims to enable fully self-generated, real-time games and experiences, although current limitations include support for only a few minutes of continuous interaction rather than extended hours.