Runway publishes research on real-time video generation

• Runway has outlined a post-training path from foundation models such as Gen-4.5 to streaming video and audio, according to Runway's announcement.
• A hand-drawn city sketch and palette become live controls in iamneubert's demo, with the animation changing as the drawing changes.
• A related interactive-world prototype couples user steering to generated sound, as c_valenzuelab's simulation demo puts it, "a real-time simulation with generated audio."
• Runway frames the work as research and gives no public availability or release date in its September 10 post, while mrjonfinger calls current real-time AI "not quite useful" for on-set visualization.

Runway's research post says its audio and video decoders stream as each latent is generated. The GWM Worlds 2 research page claims continuous 720p video at 24 fps with 48,000 Hz audio, while the Solaris paper describes a UI model that uses mouse interactions to generate its next screen state frame by frame.

Each autoregressive step takes two controls: an initial first frame and a caption supplied at the start. Runway says it retains the generated latents in context to preserve quality, then causally decodes both video and audio streams.

Drawing and palette adjustments steer the animation continuously in the sketch demonstration below. iamneubert later said the direction could "redefine what it means to generate" in a follow-up.

c_valenzuelab described the target as experiences people can interact with in real time in a post about the research. GWM Worlds 2 is the more specified version of that idea: users define five parts of a world before entering it:

• Environment
• Subjects
• Visual style
• Physical rules
• Ambience

Text actions can address a subject or the scene, alongside continuous camera motion, according to Runway's GWM Worlds 2 page. The demo applies that interaction model to a shifting 3D scene with generated audio.

Runway's technical overview describes two linked post-training stages:

1. Teacher forcing: convert a full-attention model into a temporally causal, frame-by-frame autoregressive generator.
2. Student forcing: use distribution-matching distillation to cut each frame from many denoising steps to a few.

The distillation itself has two phases:

• Off-policy: the student predicts next states from ground-truth context, guided by a frozen bidirectional teacher and a critic model.
• On-policy: the student rolls out its own latent sequence during training, so later frames are conditioned on its own earlier mistakes.

Runway says the on-policy phase produces most of the pipeline's gains. It also trains with progressively longer sequences, because small frame errors otherwise compound into visible drift over a video.

Real-time generation shifts the bottleneck to inference, Runway writes: every frame must reach the viewer quickly enough for playback while hardware is shared across sessions. The company says it is comparing quality and latency across checkpoints and hardware configurations with internal evaluation and observability tools, but the post publishes no latency measurement.

mrjonfinger's public qualifier is narrower and practical: real-time AI is still "not quite useful" for on-set visualization, though getting closer in the accompanying post.

c_valenzuelab said viewers had mistaken people in one clip for real footage, then described live action and generated material as pixels in service of story in the post. The visual comparison places raw footage beside its generated counterpart.

Solaris preceded this overview as Runway's interface world model. Its research paper says mouse actions act as conditioning signals, with each visual state synthesized autoregressively.

A clip from c_valenzuelab types "Make it rain" and then "Create a fire," prompting weather and flames inside the interface.

c_valenzuelab also observed that many first tests of frontier models try to simulate the real world in a reply.