Personalization Enablers: Training Tools

Personalization Enablers: Training Tools

Target audience: software developers

---

Training tools include five tools, including pre-training and fine-tuning models used in XR2Learn. Each tool is a modularized component with an isolated environment and dependencies that can be used separately, in combination, or as an end-to-end system (together with the Command-Line Interface – CLI).

The components are also separated by modalities, e.g., audio and bio measurements (BM) modalities. Each component is deployed using Docker to ensure easy-to-use components, reproducible development and deployment environments, and consistent results. Thus, the Training tools support cross-platform use, i.e., Windows, Linux and macOS.

All enablers are implemented using industry-standard Deep Learning frameworks (PyTorch, PyTorch Lightning) that support accelerated computing on GPUs. It is worth mentioning that all the enables are delivered as standalone units for each modality via modern containerization tools (Docker). Such a modular architecture has been proposed to facilitate and encourage open-source developers to propose and implement novel emotion recognition enablers that could be easily integrated into the proposed framework.

• Pre-processing: Pre-process raw data into an organized time window of data and labels to be used by the other components.
• Handcrafted Features Extraction: Extracts features derived from the raw data type’s properties instead of using Machine Learning for feature extraction.
• Self-Supervised Learning (SSL) Training (pre-train): Pre-train an encoder with no use of labels.
• SSL Features Extraction: Uses an encoder to generate features.
• Supervised Learning Training (fine-tuning): Trains a classification model (Enabler 4) utilizing labels.

Pre-trained encoder and fine-tuned emotion classification models are also available.

A configuration.json file is required to provide the components with the necessary specifications for running. An example example.configuration.json file is provided and can be copied and changed by the user.

Pre-processing

Component to pre-process raw data into an organized time window of data and labels to be used by the other components.

• Input: Raw files from a dataset.

Dataset Input files should be in the folder /datasets/<DATASET_NAME>, organized in different folders for each person in the dataset. For example, for the RAVDESS dataset, the folder structure should be:

.
└── datasets/
    └── RAVDESS/
        ├── Actor_01/
        │   ├── 03-01-01-01-01-01-01.wav
        │   └── (...)
        ├── Actor_02
        ├── (...)
        └── Actor_24

• Output: CSV files (train.csv, val.csv, and test.csv) and .npy files with numerical representation of each raw data input file.

Produced outputs are saved according to the following directory structure:

.
└── outputs/
    └── <dataset>/
        └── <modality>/
            ├── <process-configuration>/
            │   └── <numerical-representation-file>.npy
            ├── test.csv
            ├── train.csv
            └── val.csv

Pre-processing configuration

  "dataset_config": {
    "dataset_name": "RAVDESS",
    "number_of_labels": 8
  },

  "pre_processing_config": {
    "process": "standardize",
    "create_splits": true,
    "target_sr": 16000,
    "padding": true,
    "max_length": 5
  },