Skip to content

Latest commit

 

History

History
121 lines (99 loc) · 3.36 KB

image_classification.rst

File metadata and controls

121 lines (99 loc) · 3.36 KB

Image Classification

We provide benchmarks of different domain generalization algorithms on PACS, Office-Home, iWildCam-Wilds, Camelyon17-Wilds, FMoW-Wilds. Those domain generalization algorithms includes:

Note

DomainBed proposed three model selection methods, our hyper parameter is selected based on model's performance on training-domain validation set (first rule proposed). Concretely, we select model with highest accuracy on training-domain validation set during whole training process and use selected checkpoint to test on target domain.

Note

Different from DomainBed, we do not freeze BatchNorm2d layers and do not insert additional Dropout layer except for PACS dataset. Besides, we use SGD with momentum as default optimizer and find it usually achieves better results than Adam. During training, a cosine learning rate decay strategy is used.

Note

  • ERM refers to the model trained with ERM, which is a strong baseline.
  • Avg is the average accuracy.
  • Acc1 is the top-1 accuracy on OOD test set for Wilds datasets.

PACS accuracy on ResNet-50

Methods avg A C P S
ERM 86.4 88.5 78.4 97.2 81.4
IBN 87.8 88.2 84.5 97.1 81.4
MixStyle 87.4 87.8 82.3 95.0 84.5
MLDG 87.2 88.2 81.4 96.6 82.5
IRM 86.9 88.0 82.5 98.0 79.0
VREx 87.0 87.2 82.3 97.4 81.0
GroupDRO 87.3 88.9 81.7 97.8 80.8
CORAL 86.4 89.1 80.0 97.4 79.1

Office-Home accuracy on ResNet-50

Methods avg A C P R
ERM 70.8 68.3 55.9 78.9 80.0
IBN 69.9 67.4 55.2 77.3 79.6
MixStyle 71.7 66.8 58.1 78.0 79.9
MLDG 70.3 65.9 57.6 78.2 79.6
IRM 70.3 66.7 54.8 78.6 80.9
VREx 70.2 66.9 54.9 78.2 80.9
GroupDRO 70.0 66.7 55.2 78.8 79.9
CORAL 70.9 68.3 55.4 78.8 81.0

iWildCam-Wilds accuracy on ResNet-50

Methods acc1
ERM 75.4
IBN 77.3
MixStyle 71.0
IRM 75.5
VREx 71.5
GroupDRO 28.0
CORAL 71.0

Camelyon17-Wilds accuracy on ResNet-50

Methods acc1
ERM 94.6
IBN 96.1
MixStyle 94.2
MLDG 91.2
IRM 94.9
VREx 88.2
GroupDRO 93.1
CORAL 90.6

FMoW-Wilds accuracy on DenseNet-121

Methods acc1
ERM 53.0
MLDG 47.4
IRM 48.1
VREx 50.4
GroupDRO 47.5
CORAL 50.0