Description

Source Motion with Sound

A standing-up motion with recording of my voice describing motion.

Target Motion

Another standing-up motion.

Synthesis Constraints

Automatically defined by motion matching algorithm.

Comments

The input soundtrack plays a different sound for sitting, standing-up and walking motions. Given a new standing-up motion, we want the same sounds sounds to play for the same motions. Our algorithm automatically defines constraints for the synthesis to achieve this effect.

The algorithm works by first segmenting the source and target motion into fragments. We obtain the 3D barycentre of the figure at each frame. We then detect the minima of the norm of the motion of the 3D barycentre. These define the segmentation boundaries of the fragments. This is done over both the source and target motions.

We then calculate the distance between each source fragment to every other target fragment. The top-3 best source matches for each target fragment are kept. We then find a path that maximizes the use of consecutive source fragments. The audio portion assigned to each source fragment is then assigned as a constraint to each target fragment. The synthesis finally synthesizes the resulting soundtrack.

 

 

 

Self-Similarity Synthesis

 

Source Motion with Sound

 

 

 

Shows motion of the 3D barycentre of the character for the Source motion.

Shows the norm of the 3D barycentre of the character for the Source motion. The minima are selected as fragment boundaries. The minima correspond to stoppages in the motion, which are typically good markers of changes in the motion.

Shows motion of the 3D barycentre of the character for the Source motion, along with selected minima.

Target Motion

 

 

 

Shows motion of the 3D barycentre of the character for the Target motion.

Shows the norm of the 3D barycentre of the character for the Target motion. The minima are selected as fragment boundaries. The minima correspond to stoppages in the motion, which are typically good markers of changes in the motion.

Shows motion of the 3D barycentre of the character for the Target motion, along with selected minima.

 

 

New Sound

 

(top) Shows the segmented Source soundtrack along the dynamically generated constraints source segments. (middle) Shows the synthesized sound for the Target motion. (bottom) Shows Constraint targets along with the origin of the grains in these targets regions.

Shows the Euclidean distances between the each fragment of the Source motion with that of the Target motion's.