The choreography 2 dataset

The data presented in this page can be downloaded from the public repository

If you use this database in your experiments, please cite it (DOI:10.5281/zenodo.29551) or the following paper:

Mangin, P.Y. Oudeyer, Learning semantic components from sub symbolic multi modal perception to appear in the Joint IEEE International Conference on Development and Learning an on Epigenetic Robotics (ICDL EpiRob), Osaka (Japan) (2013) (More information, bibtex)


This database contains choreography motions recorded through a kinect device. It contains a total of 1100 examples of 10 different gestures that are spanned over one or two limbs, either the legs (e.g. walk, squat), left or right arm (e.g. wave hand, punch) or both arms (e.g. clap in hands, paddle).

Each example (or record) contained in the dataset consists in two elements:

Description of the data

The data has been acquired through a kinect camera and the OpenNI drivers through its ROS <> interface, which yields a stream of values of markers on the body.

Each example from the dataset is associated to a sequence of 3D positions of each of the 15 markers. Thus for a sequence of length T, the example would corresponds to T*15*7 values.

The position of the following list of markers was recorded:

head, neck, left_hip, left_hip, left_shoulder, left_elbow, left_hand, left_knee, left_foot, right_hip, right_shoulder, right_elbow, right_hand, right_knee, right_foot, right_hand

A list of gestures and their descriptions can be found at the end of this document.


This data is accessible in three data formats:

###The text format

The set of examples consists in:

These are distributed in a compressed archive (tar.gz).

An example of a json file is given below. They all have a similar structure.

"marker_names": [
"data_dir": "mixed_partial_data", 
"name": "mixed_partial", 
"records": [
    "data_id": 0, 
    "labels": [
    "data_id": 1, 
    "labels": [

It contains the following data:

For each record listed in th json file there exists a text file in the ‘data_dir’ directory, which name is the ‘data_id’ plus a ‘.txt’ extension.

The text files contains the sequence of positions of the marker. Each set of values at a given time is given as a line of space separated floating numbers (formated as ‘5.948645401000976562e+01’).

Each line contains 7 successive values for each marker which are there 3D coordinates together with a representation of the rotation of the frame between previous and next segment. The rotation is encoded in quaternion representation as described on the ROS time frame page . Thus each line contains 7xM values with M the number of markers.

The numpy format

In this format each set of examples is described by two files: a json file and a compressed numpy data file (.npz).

The json file is very similar to the one from the text format, the only difference is that the ‘data_dir’ element is replaced by a ‘data_file’ element containing the path to the data file.

The data file is a numpy compressed data file storing one array for each example. The name of the array is given by the ‘data_id’ element. Each data array (one for each record) is of shape (T, M, 7) where T is the length of the example and M the number of markers.

The following code can be used to load a set of example in python.

import os
import json

import numpy as np

FILE = 'path/to/mixed_full.json'

with open(FILE, 'r') as meta_file:
    meta = json.load(meta_file)
    # meta is a dictionary containing data from the json file
path_to_data = os.path.join(os.path.dirname(FILE), meta['data_file'])
loaded_data = np.load(path_to_data)
data = []
labels = []
for r in meta['records']:
    data.append(loaded_data[str(r['data_id'])]) # numpy array
    labels.append(r['labels']) # list of labels as integers

print "Loaded %d examples for ``%s`` set." % (len(data), meta['name'])
print "Each data example is a (T, %d, 3) array." % len(meta['marker_names'])
print "The second dimension corresponds to markers:"
print "\t- %s" % '\n\t- '.join(meta['marker_names'])
return (data, labels, meta['marker_names'])

The Matlab format

In the Matlab format, a set of examples is described by a single ‘.mat’ file containing the following elements:


For more information, feel free to contact me at olivier.mangin at inria dot fr.


###List of gestures

A table with illustrations of gestures is presented in the gesture illustration page. .

Id Limb(s) Description
1 both legs squat
2 both legs walk
3 right leg make a flag/P on right leg
4 both arms clap
5 both arms mimic paddling left
6 right arm mimic punching with right arm
7 right arm right arm horizontal goes from side to front
8 left arm horizontal left arm, forearm goes down to form a square angle
9 left arm make waves on left arm
10 left arm say hello with left arm