Languages

TPT-Dance&Actions : a multimodal corpus of human activities

Aymeric Masurelle, Ahmed Rida Sekkat, Slim Essid and Gaël Richard

! UNDER CONSTRUCTION ! ! UNDER CONSTRUCTION ! ! UNDER CONSTRUCTION !


Introduction:

We present a new multimodal database of human activities, TPT – Dance & Actions, that focuses on 14 dance choreographies and 13 sequences of other human activities:

  • Dance scenes:
    • 8 Lindy-Hop choreographies performed by 7 dancers (clh1->clh8),
    • 3 Salsa choreographies performed by 10 dancers (css1->css3),
    • 3 Classical dance choreographies performed by 3 dancers (cdc1, cdc3 and cdc5).
  • Fitness scenes cadenced by music:
    • 8 fitness choreographies performed by 16 participants (cfs1->cfs4 and cfq1->cfq4).
  • Isolated actions:
    • 5 isolated action sequences performed by 16 participants (cgs1->cgs5).

Those different scenes have been recorded using a variety of sensor arrays: RGB cameras, depth cameras, microphones, inertial sensor suit and piezoelectric sensors on the dance floor.

Overview of the multimodal system used for recording the data of our corpus.

Overview of the multimodal system used for recording the data of our corpus.


Dataset:

  • File structure:

The present dataset has the following file structure:

./<type_of_gesture_scene>/p<participant_index>/<short_name_of_the_sequence>_<file_type_name>_t-<take_number>.<file_extension>

For examples:

./salsa/p28/css1_cam01_t-00.mkv  (for camrecorder files)

./salsa/p28/css1_kinec01_t-00.h5 (for kinect rgb+depth files)

./salsa/p28/css1_kinect03-micro02_t-00.wav (for kinect audio files)

./salsa/p28/css1_micro01_t-00.wav (for microphone audio files)

./salsa/p28/css1_piezo01_t-00.wav (for piezo audio files)

./salsa/p28/css1_xsens_t-00.mvnx (for xsens files)

  • Annotations:

For the different gesture sequences, an annotation file (.txt) is available. Each action, Fitness gesture and dance step is named and its ideal temporal limits are expressed in SMPTE time format:

<SMPTE_begin_gesture>, <SMPTE_end_gesture>, <gesture_name>

For the different sequences, the annotations indicate the gesture label in respect to vocal indications and/or musical beat in the case of Fitness and Dance.

  • Availability:

These data are available to download for research purpose. In order to get it, please contact ….@……

If you use them to publish results, please cite reference [1].

 


Data synchronisation:

Overview of the synchronisation strategy of the different modalities between them

Overview of the synchronisation strategy of the different modalities between them.

The above figure presents an overview of the synchronisation strategy between the different data modalities. The following subsets of data are already synchronized:

  • the subset M1 consists of 4 audio streams from schoeps microphones synchronised between them,
  • the subset P1 consisting of 4 audio streams from piezoelectric sensors synchronised with each other,
  • the subsets KA1, KA2, KA3 and KA4, each one composed of four synchronised audio streams from the microphones of each kinects,
  • the subsets KV1, KV2, KV3 and KV4 formed of synchronised video streams and depthmaps of each kinects,
  • the subsets C1, C2, C3 and C4 consists of video and audio streams all synchronised together,
  • the subset X1 composed of synchronised inertial data and major joint positions.

Thus, to achieve a global synchronisation of all data streams together, it is sufficient to synchronize a data stream of each subset with a data stream of another subset.

For more details about this global synchronisation process, please check reference [2].


References:

[1] TPT-Dance&Actions : un corpus multimodal d’activités humaines,

Aymeric Masurelle, Ahmed Rida Sekkat, Slim Essid, Gaël Richard,

Revue Traitement du Signal, 2015 (accepted).

[2] Analyse multimodale de scènes de danses,

Ahmed Rida Sekkat,

Master thesis, 2015.

 


Acknowledgements:
The authors thank warmly all the persons who have been contributing to the creation of this corpus, and especially:

  • participants: Gaël, Bertrand, Anaïs, Ariane, Stéphane, Laurent, Eric, Sylvie R., Rachel, Roland, Floriane B., Zhoungwei, Youcef, Damien, Floriane D., Eve-Marie, Pascal, Sylvie M., Juliette, Albane, Xabier, Hequn, Caroline, Anne-Claire, Alexis, Clément, Nicolas, Frédéric, Thomas, Paul R., François, Paul M., Romain, Simon L., Victor, Mathieu, Nesrine, Florian, Brian, Simon D., Chloé, Alice, and,
  • technicians: Rida, Slim, Stéphane, Gilbert, Michel, Nesrine, Mounira.

This work was supported by the European Commission under contract “FP7-287723 REVERIE”.

Comments are closed.