Image-to-video generation · 3D motion control

Generative
Cinematographer

Composing Camera and Object Motion in 3D

Start with one image. Choreograph the camera and subjects in a shared 3D world.
Turn your motion into a generated video.

Choose an exampleViewing 1 / 5

Click a scene to switch the image, 3D controls, and generated video below.

01 One input image
Input image for the selected scene
02 Authored 3D motion
03 Generated video · Ours

Single-image scene scaffoldIndependent local handlesJoint camera & object control
01 / The idea

Compose a shot.
In the world, in 3D.

A movement on the screen can come from the subject, the camera, or both. A 2D trajectory alone leaves that relationship ambiguous.

We lift an image into a colored point cloud and give artists one shared coordinate system for the camera and foreground. The video model then turns this geometric guidance into a coherent shot.

01

One world for camera and object motion

Draw a camera path and move selected foreground regions independently. Both controls describe motion in the same 3D scene.

02

A few handles, many possible motions

Each handle moves a selected group of points together. Several local handles can approximate articulated or non-rigid motion, without a skeleton or a physics simulator.

03

Position and identity, kept together

Three rasterized guidance maps tell the model where the background is, where a foreground handle is now, and which handle it is across frames and viewpoints.

02 / The method

From one image
to a choreographed video.

Author in 3D, render the motion into image-plane maps, then condition a pretrained video generator. The controls carry geometry; the model supplies appearance and motion between them.

01 / RECONSTRUCT

Lift the image

Estimate depth and a dynamic mask. Back-project pixels into a colored point cloud, using the first camera frame as the world frame.

02 / CHOREOGRAPH

Place motion handles

Select local foreground regions in a 3D interface such as Blender. Author their trajectories and the camera path.

03 / RASTERIZE

Render three maps

Project the static background and moving handles. Encode world XYZ and persistent foreground identity, with validity masks.

04 / GENERATE

Guide video diffusion

A frozen Wan VAE encodes the maps. A trainable side branch, residual projections, and LoRA adapters condition generation.

The framework figure from the current paper. Select it for a closer look.Vector PDF ↗
WHAT LEARNS?

The Wan VAE and base diffusion weights stay frozen. The lightweight side branch, cross-branch convolutional projections, and main-branch LoRA adapters are trained. The side branch operates at half the main branch’s spatial resolution and injects geometric features through residual connections.

03 / An interactive explanation

Same handle.
New position. New viewpoint.

Move the camera, then move the handle. Watch the projected maps change. The identity color stays attached to the handle; its XYZ color changes only when its world position changes.

Explore the correspondence maps

Conceptual illustration · no model inference

Shared 3D scene Fixed overview
Static backgroundMotion handleCamera

Background XYZ

Fixed world positions. The same points keep the same colors across views.

World reference

Foreground XYZ

Current world position. The color changes when the handle moves.

RGB

Foreground identity

A fixed base color. It identifies this handle throughout the shot.

Handle 01

Try the camera slider: projections move, but the world-coordinate colors stay the same.

A closer look at the representation

Both XYZ maps use a per-axis normalization derived once from the first-frame scene: bounding-box center o and half-range σ. Normalized X, Y, and Z become red, green, and blue. The same normalization is used in every frame.

RGB = ½ [(p − o) / σ + 1]

A projected pixel says where the point appears; its XYZ value says where it is in the world. Because a moving point changes XYZ, the separate foreground identity map supplies its persistent handle label. This allows the model to interpret foreground motion relative to the static scene.

Swipe or scroll sideways to inspect the full diagram. Select it to enlarge.

The paper’s bus example: background points preserve their world-coordinate colors while the moving bus changes position.Vector PDF ↗
04 / Side-by-side comparisons

One scene.
Every method, together.

Compare our result with seven baselines in one view. Browse the scenes, inspect the intended motion, and replay all outputs together.

Input image
Selected scene input
Scene detail
Authored motion

05 / Generated results

Different scenes.
One control language.

Explore 13 selected examples from the supplied gallery. Each result follows an authored motion. Open a video to inspect it, or choose “Compare” to see its controls and baseline outputs.

06 / Scope & limitations

Geometry provides control.
The scaffold has limits.

One view is incomplete

Monocular depth errors and unseen surfaces can produce geometric inconsistencies, especially under large viewpoint changes, reflections, and thin structures.

Local rigidity is an approximation

Handles are better suited to local motion than to tearing, fracture, or fluid dynamics. Fine deformation can require many handles, increasing the authoring effort.

Control does not enforce physics

Independent trajectories do not impose physical or causal constraints. XYZ maps also use a fixed first-frame normalization range, which can be exceeded by large edits.

Continue exploring

See the paper. Explore the motion.

Project page accompanying the current manuscript snapshot · September 2026