Rhythm of Motion Extraction and Rhythm-Based Cross-Media Alignment for Dance Videos

Wei-Ta Chu and Shang-Ying Tsai

Multimedia Computing Laboratory
Dept. of Computer Science and Information Engineering
National Chung Cheng University


1. Introduction

We present how to extract rhythm information in dance videos and music signals, and accordingly correlate two media based on this novel representation. From danceríŽs movement, we construct motion trajectories, detect turnings and stops of trajectories, and then estimate rhythm of motion (ROM). For a music signal, music beats are extracted based on integration of audio dynamics in different frequency bands. Two modalities are therefore represented by sequences of rhythm information to facilitate determination of cross-media correspondence. Two applications are developed to show the feasibility of utilizing cross-media correspondence íV background music replacement and music video generation. In the experiments, we evaluate performance of ROM extraction, and conduct subjective/objective evaluation to show that rich experience can be obtained by the proposed applications. We also suggest that rhythm information may be used in more multimedia content applications, such as rhythm-based multimedia retrieval.

2. Evaluation Dataset

Note that the data provided here may be used freely for research purposes but it cannot be used for commercial purposes.

We provide videos with Audio Video Interleave (AVI) Format.
Encoding settings -
Video: MPEG-4 Video 320x240 (4:3) 30.00fps 232~247Kbps
Audio: MPEG Audio 48000Hz stereo 320Kbps
Recommended player: Cyberlink PowerDVD or media player classic (download)

 

3. Background Music Replacement (Dataset2)

4. Music Video Generation

5. Citation

W.-T. Chu and S.-Y. Tsai, "Rhythm of Motion Extraction and Rhythm-Based Cross-Media Alignment for Dance Videos," IEEE Transactions on Multimedia, vol. 14, no. 1, pp. 129-141, 2012.


Any problem please contact .