Industry Safety Beacon Application
Synthetic Dataset Generator is a unity project that generates annotated RGB images for the purpose of training object detection algorithms. The scenario that gets examined by this application is the detection of hazards inside industrial environments through training machine learning components to recognize whether the appropriate safety equipment is worn.
Through the Unity game engine, 3D models of working employees equipped with different combinations of safety wearables are placed in various poses around the capturing frame. The resulting composition is being saved as a PNG accompanied by a JSON file containing annotations about the objects of the image. The dataset creation workflow will be broken down into different parts below in order to be better understood and utilized in scenarios other than hazard detection.
The project is built on top of the Perception package, available from the Unity Asset Store. We exploit the option for labelling 3D models, injecting custom randomizing scripts and exporting RGB annotated images that this package offers. If you want to run Beacon Application 3 or to develop a similar application, make sure the Perception package is properly included inside your Unity project.
Large models of the project are available here. Download the three .fbx models and place each one inside path: Assets/Models/[Νame_Οf Μodel].fbx
The scene that we run to create the dataset is located inside the Scenes folder. This is the correct directory to place any new scene that corresponds to the creation of a different dataset type. Currently, inside this directory there is only this one scene active scene file.
The orchestration of all the procedures that take place according to given configuration settings is the WearablesScenario game object to which a Fixed Length Scenario component is attached. Through this component it gets decided how many iterations will take place during runtime and which randomizing script will be active. All custom randomizing scripts are located inside the Scripts directory, but we will dig into the concept of randomizing scripts later.
The models that are being instantiated during the creation process are located inside the Models directory. We are working with 3 different human models that contain the available safety wearables inside the model file (in FBX format). This approach, instead of loading each 3D model of the equipment one by one, was chosen because it allows for personalized modifications to be applied to the wearables giving a more realistic result when worn by each human figure.
Out of every human model, we created prefabs, to represent the variation of their body pose directly thought Unity and stored those in inside the Prefabs directory.
The color variations of the safety equipment demand access to a pool of textures, created in our case manually using photo editing software, more specifically GIMP. At runtime, randomizers assign a texture to the material of every wearable that is intended to change color. For this to be achieved, wearables attached to each human figure have their own personal material instead of sharing one, to avoid capturing identical objects during sampling. However, in cases of objects that do not participate in this procedure, some materials are being shared, for memory efficiency. This process led us to separate textures from materials and store them in different directories.
We are using perception’s labeling component for the desired categories that we want the object detection algorithms to learn.
When this component is attached to a game object, spatial information about it from the output RGB sample gets recorded in the annotation file along with its label name and ID. In our example, we only use information about the 2d bounding boxes (center, width, height) but the choice depends on the type of information that your machine learning algorithm uses.
Other information provided by Perception includes Depth Labeling, Instance Segmentation Labeling, Semantic Segmentation Labeling etc.
Controlling the exported information type is done via the Perception Camera Script that needs to be attached in the scene’s main camera. The script allows for more than one type of information can be extracted together in a single run
To train the object detection algorithm successfully and reach comprehensive results we need to provide sufficient data that is also diverse, resilient, and representative. To achieve this, we determined 8 different layouts for our created scene that work as grids which contain predefined positions for the 3D human objects to spawn.
Each placeholding position constrains the scale and rotation of the human model that will take over this position inside a configurable limit and the selection of the model is happening randomly. Random is also the selection of the equipment, and its color, provided to a human object, the posture of the model and its rotation and scale (always within the configured limits).
This way we control how random the results will be, and we depict equipped human models in more natural positions. Those 8 different layouts succeed each other in a rotational order until the dataset creation process is finished.
This scenario is developed by a combination of custom randomizing scripts attached to game objects of the scene, to orchestrate the layout system and give the desired behavior to each one. However, keep in mind that different requirements for your dataset may need different scenarios and concluding to one is a process of continuous testing and refinement of your developed scene
The configuration of the perception’s scenario component decides which randomizers will be active at runtime and the order of their execution. Randomizers are scripts provided by the package, that in every iteration of the program apply their functionality to a set of game objects that get defined using randomizer tags or placeholders (for prefabs).
Attaching a randomizer tag to a game object gives access to the randomizing script to modify its properties in various ways such as: change its quaternion, spawn it to a random position, create background noise, change scene lighting etc. For more information about randomizers, you can read the Perception’s documentation.
In cases where a specific functionality is needed that is not provided by the built-in randomizers, there is the option to create your own. In our application, we used our own scripts to manage the spatial distribution of our models, the way they rotate and change colors. The table below describes the functionality of every custom script created for this purpose.
It is attached to the Grid game objects, making it behave as an orchestrator of the available layouts. Through it, a time interval is configured. When a time interval is passed the script iterates through its child objects, and randomly enables one of them utilizing the exposed methods of the Grid Enabler script. The children of the Grid game object represent all the available layouts of the scene
Besides exposing functionality to enable/disable the Grid game object, it provides layout functionality to every game object it is attached to. The enabled layout activates its children, that work as fixed placeholding positions.
Attached to every child of a Grid Enabler, it renders a different human model in each iteration, when it is active. This script lets you configure if you want to define limit values for the scale and rotation of its models. An additional functionality, to configure the probability of scale and rotation to not be randomized is implemented.
Attached to every rendered human model. It randomizes scale and rotation of the model respecting the limits set by the Seat Picker
It separates wearables into different categories, in our case in Helmets, Vests, Glasses and Gloves. The game objects that represent those wearables can be dragged and dropped into those categories, so that the model is completely customizable. The script randomly picks one object of each category to be rendered. In our case we also provide a blank game object for each category for the option of no equipment.
It accepts different color options and applies a random one to the game object it is attached to in every iteration. Hue Randomizer is used to change the color of the wearables.
It applies a different random texture to the game object it is attached to. Its functionality is similar to the Hue Randomizer’s but is useful for elements that require different materials instead of different colors. The pool of the available materials for each instance must be an existing directory of the project and the selected directory is configurable.
This repository provides all the essential tools and procedures required to prepare and merge the synthetic and CHV dataset into a format compatible with the YOLOv5 model for training purposes.
It serves as a proof of concept, demonstrating the potential of achieving excellent results by combining a small subset of real images with a larger synthetic dataset.
The subsequent steps are aimed at preparing the dataset for training an object detection algorithm, specifically designed to recognize and classify person, vest, and helmet categories.
To effectively train an object detection system using the YOLOv5 model, the dataset should include both the images and corresponding text files that provide annotations for the objects present in these images. For detailed insights into the data structure and annotation format, please refer to the information available at this link.
The synthetic dataset includes labels for various classes as listed below:
1: gloves
2: glasses
3: helmet
4: vest
5: person
Annotations labeled with 1 correspond to gloves, those with label 2 represent glasses, and so on for the other categories.
- To prepare your dataset annotations in the format required by the model, execute the main.py script located in the dataset_formatter directory by running the following command in your terminal:
python dataset_formatter/main.py
Upon execution, the script will prompt you to select the directory containing your synthetic images. Navigate to and select the location of your synthetic images to proceed with the formatting process.
- This guide focuses on training an object detection algorithm to identify person, vest, and helmet. Therefore, we will remove the classes for gloves and glasses from the dataset:
python utils/delete_lines.py /path/to/synthetic_dataset 1 2
- Last by running the following script all the folders of the images and the labels will be organized in the necessary format:
./create_synthetic_dataset.sh
Upon executing the script, you will be prompted with the following message:
"Enter full path of the synthetic dataset:"
Enter the complete path to the synthetic dataset to initiate the preprocessing process.
After running the bash script, the labels corresponding to each class will be updated as follows:
0: person
1: vest
2: helmet
All the necessary preprocessing for the synthetic dataset is done!!!
If your intention is to train on different classes, such as including all classes or just two classes like gloves and person, please be aware that this will require adjustments to both the files provided in the "utils" folder and the bash scripts.
The Color Helmet and Vest (CHV) dataset is a dataset that contains real rgb images tailored for the detection of individuals wearing vests and helmets in four distinct colors.
Here are the labels and their corresponding classes:
0: person
1: vest
2: helmet blue color
3: helmet yellow color
4: helmet white color
5: helmet red color
First download the dataset by following this link
Once downloaded, execute the provided bash script to prepare the dataset:
./create_chv_dataset.sh
Following the script's execution, you will need to input both the full path to the CHV dataset and the location where the processed dataset should be saved. In the location to save the dataset you also need to add and the name of the saving folder in the end of the path insertion. For example "path/to/save/the/dataset/name_of_the_dataset"
After running the bash script, the labels corresponding to each class will be updated as follows:
0: person
1: vest
2: helmet
All the necessary preprocessing for the CHV dataset is done!!!
If you wish, you have the option to test the annotations within the images by executing the following command:
python utils/test_annot.py path/to/dataset CHV 2
The test_annot.py script accepts three distinct arguments:
- Path of the dataset
- Type of the dataset (synthetic or CHV)
- Label to be tested: 0 or 1 or 2
Here are some examples of the annotations of the helmet class(label number 2) of the CHV dataset:
To combine the datasets run the following bash script:
./create_combined_dataset.sh
The script will ask the full path of the synthetic dataset and the full path of the CHV dataset.
Upon running the bash script, a new folder named combined_dataset will be generated. Inside this folder, you will find:
- A test dataset consisting exclusively of 133 real images from the CHV dataset. This selection ensures testing of the model's real-world applicability.
- In the train folder, there will be a total of 50 images from the CHV dataset and 600 images from the synthetic dataset, facilitating comprehensive training.
- The validation dataset will encompass 25 images exclusively from the CHV dataset, ensuring a diverse and balanced dataset for model evaluation.
If you want to use different number of images of each dataset you must change the values (NUM_OF_REAL_IMAGES, NUM_OF_VALIDATION_IMAGES, NUM_OF_SYNTHETIC_IMAGES) in the create_combined_dataset.sh file.
The dataset is complete and ready for training!!!
To train the YOLOv5 model, follow these steps:
- Clone the official repository and install the required packages:
git clone https://github.com/ultralytics/yolov5 # clone
cd yolov5
pip install -r requirements.txt # install
- Navigate to the 'data' folder and create a YAML configuration file:
cd data
touch syn_chv.yaml
- Open the yaml and add the following content:
path: path/to/dataset # add the full path of the dataset
train: images/train
val: images/test
test: images/test
# Classes
names:
0 : person
1 : vest
2 : helmet
- Start training the model by running the following commands:
cd ..
python3 train.py --epochs 10 --data syn_chv.yaml --weights yolov5s.pt
That's all !! The training process is finished.
Utilizing the trained model, you can now create a real-time detection to check whether a person is equipped with helmets or vests. This process simply requires a camera to be operational.
To initiate live inference, execute the following commands:
cd ..
mv hazard_detection.py path/of/yolov5_repo
cd path/of/yolov5_repo
python hazard_detection.py --weights /path/to/weights/best.pt --source 0 --conf-thres 0.70
In the event that the camera fails to recognize the subject, consider altering the --source parameter to a different value, such as 1 or 2.
The hazard_detection.py script is designed to identify individuals who are either wearing or not wearing the necessary safety gear. In cases where the appropriate gear is missing, an alert will be displayed in the terminal.
|
Wiki - Immerse yourself in the world of XR2Learn |