Animation / Video to 3D Scene

Estimated reading: 5 minutes

Introducing the “Animation / Video to 3D Scene” project type, where it is possible to create a complete CG scene in a few simple steps. Work with a sequence of shots, contained within one scene, and get a fully aligned animated sequence ready to be placed in any environment. Your CG scene will retain all of your shots oriented in space, preserving sequence continuity, with correctly oriented characters and cameras, ready for further editing.

This project type works best on a sequence of shots.
It is assumed that the sequence takes place in one scene.

The “Animation / Video to 3D Scene” process consists of 4 main steps:

Animation / Video to 3D Scene Project Guidelines

Rendering

Animation / Video to 3D Scene project type uses a rasterConverts 3D models into a 2D pixel grid. Resources: Rendering
3D Rasterisation render engine.

Review the Shading section of the Character Setup documentation for information on how to prepare your character's shaders for this project type.

The “Animation / Video to 3D Scene” project type will align shots to one another, essentially converting your video into a continuous 3D scene.

It will additionally try to fit the aligned sequence to the environment if it is selected.

Here is what you need to know to make the most out of the “Animation / Video to 3D Scene” project type’s features.

Please avoid using fractional framerates when shooting input videos.

Currently, fractional framerates can cause animation drift.

One Scene Per Project

The “Animation / Video to 3D Scene” project type is meant to work with one continuous sequence that takes place in one scene.

Establishing Shot

It is important to have an establishing shot where many/all actors will be visible. This shot will later help orient other shots.

Actors and the camera do not need to be stationary in this shot as long as sufficient attention is placed on spatial continuity.

Continuity

Pay attention to continuity. Continuity in filmmaking refers to the consistent presentation of story elements across successive shots and scenes, ensuring that there are no noticeable inconsistencies in the position, actions, or appearances of characters and objects.

Take extra care that actors start and stop in consistent positions and orientations between shots to increase spatial continuity. Filming a scene as if it is one continuous action would help enormously, as the system will be able to position shots more clearly in space.

Framing and Filming

Avoid using over-the-shoulder shots, as they often complicate tracking and can lead to inaccuracies in the final footage. Similarly, minimize occlusions in your scenes, as they significantly increase the risk of tracking failures due to obscured or missing visual data.

The project functions optimally on flat ground planes; significant elevation changes may result in less accurate outcomes.

Results are very sensitive to focal length adjustments. Thus, if possible, always manually enter the correct focal length used during shooting to minimize sliding of the feet.

Camera Tracking

Steer clear of uniformly colored or textureless backgrounds as they pose challenges for accurate camera tracking.

The project may face limitations with overcrowded videos, particularly when dynamic objects dominate most of the frame (cars, people, etc.).

Avoid filming with rapid camera movements that may challenge the tracking system’s ability to accurately follow, as well as variable camera zooms, which can similarly complicate the tracking process.

Motion prediction

Motion Prediction improves the pose estimation when a subject is partially or fully occluded or has body parts out of the camera frame. In order to get the most from our model you should follow these guidelines:

Currently only available on Animation / Video to 3D Scene project type
Use static camera – steady handheld shots also work.
Use flat ground – avoid stairs, hills or any multi-level areas.
30 fps videos work best – avoid high frame rate or slow-motion clips.
Avoid crowds – one or two non-overlapping characters is preferred.
Avoid shooting actors in a close-up view where only their chest and above is visible.

Character Compatibility

Compatibility

To ensure your character assets will be handled correctly for this project type, please make sure that you validate your characters with the “Enable USD Support” flag selected in the validation add-ons.

Characters will receive baked animation on all their bones.

Shading

Standard PBR workflow shaders are supported. Only image texture nodes are supported.

Transparent objects, such as eyes, should use Opacity channels.

SSS is supported over standard SSS channels.

3D Scene

Scene Alignment

All shots in the project will be processed and aligned to one another, creating a scene with continuous action, animation, and camera track.

Export scenes will contain aligned shots, animated characters, and a camera track.

As in the Live Action project type, character textures are not provided with the export element and will have to be loaded into the scene separately.

In the case that the environment is chosen, the asset will have to be downloaded separately and added to the scene.

Environment Alignment

If an environment is chosen, the aligned scene will be fitted into the environment based on its footprint. Camera motion also contributes to the footprint, as does any actor’s motion through the scene.

The alignment will try to avoid any obstacles in the scene.

It is recommended that the scene measurements be referenced and that the filmed material be kept contained within those measurements.