ViT-S and drop_path_rate

[vladchimescu]

Hi,
I'm trying to train DINOv2 with the ViT-S backbone. I've noticed that the model training crashes if drop_path_rate > 0. In particular, I get the following error

RuntimeError: Expected source.size(1) % kNumElementsPerBlock == 0 to be true, but got false.  (Could this error message be improved?  If so, please report an enhancement request to PyTorch.)

with the traceback going all the way to line 173

As recommended in issue #108, one need to set drop_path_rate = 0 when using the ViT-S backbone. What is the reason for this behaviour? In the original DINO, the drop_path_rate = 0.1 for both ViT-S and ViT-B.

[qasfb]

Sorry for the late reply; if this problem persists, does this PR fix it for you by any chance ? #195

[josephcappadona]

Yes, that worked for me, thank you! Except I had to add back the fmha import since it seems to have been mistakenly removed from the updated import statement.

[mehrnoosh-afshar]

Hi I have the same issue with drop_path_rate>0 for VIT-SMALL, can you let me know what is the reason and which commit you fix this issue? I am using the latest version of repo so far and issue still exist apperantly

[natwille1]

Hi I have the same issue with drop_path_rate>0 for VIT-SMALL, can you let me know what is the reason and which commit you fix this issue? I am using the latest version of repo so far and issue still exist apperantly

same here!