how can i evaluate when I have class IDs in ground truth annotations that are different to the class IDs in predicted annotations ?

[Naren83]

Hello @rafaelpadilla ,

My Gound truth annotations are in yolo format like this 0 0.4359375 0.52109375 0.3375 0.95390625 with class IDs of 0,1,2 with 0 being door , 1 being chair and 2 being table.

I have trained a YOLO v5 model,got the predicted annotations and I used your repo to perform the evaluation metrics and I got the metrics for the all the three classes and the mAP.

but, I want to compare my model with the state of the art models and I need to calculate the evaluation metrics for all the state of the art models. For example I want to compare my model with the state of art model Yolo v5l. I took the weights of yolov5l and performed the detections on my test images and I got all the annotations for them.

The problem here is the yolov5l is trained on the coco metrics and the predicted annotations class lDs are different to the class IDs I have for my test images which are the ground truth annotations, so I couldnt perform evaluation metrics using your repo.

for example the ground truth annotation for a test image is 0 0.3515625 0.49609375 0.246875 0.49765625
0 0.45 0.65078125 0.36875 0.69765625

The predicted annotation for the same image using the weights of yolov5l is 56 0.87618 0.45 0.6125 0.365625 0.58125
56 0.907485 0.346875 0.465625 0.253125 0.428125

Can you please help me solve this ?

[rafaelpadilla]

Hi @Naren83 ,

The ids are mapped differently. First, you will need to find the correct mapping between "your dataset" -> "COCO's id":
Then, you will need to change all detection files using the corresponding COCO's id.

Let's say you mapped 0 (door) as being 71 in COCO. Then you need to change all your files accordingly. For example, your file my_image.txt which shows

0 0.3515625 0.49609375 0.246875 0.49765625
0 0.45 0.65078125 0.36875 0.69765625

will have to be changed to:

71 0.3515625 0.49609375 0.246875 0.49765625
71 0.45 0.65078125 0.36875 0.69765625

Another alternative would be to run YOLOv5 in inference mode changing the ids in your yaml, so it can match your 0, 1 and 2 classes.

Note that this may not be a fair comparison, as you may be comparing your model against another (more robust one) that was trained with all COCO classes, and, thus, was trained to distinguish better many other objects classes.

Hope that helps

[Naren83]

Hello @rafaelpadilla Thanks for the information . I have a dataset consisting of doors , chairs and tables which are collected within our university premises using a robot. i found out that the YoloV5 model was trained on COCO dataset and it doesnot contain doors labels, so i want to show its better to train a custom model for certain domain specific cases instead of the pre trained models and show case that my custom model performed well on the dataset instead of the pre trained models. As you gave me an idea about to change the id's, my problem here is -

After runing the inference on my dataset with the yolov5l and yolov5m pre trained weights, it didnt detect any doors and I dont have any information about the doors as it was not pre trained on door class label, I have just information about chairs and some other class labels it detected such as mnouse and keyboard which are irreravalnt to me. Can you give me an idea about how to proceed it further ?