Expected time/epoch for conceptual captions (R50) #55
Replies: 3 comments 1 reply
-
When I run on a machine with 8xV100 using the settings from the README, it takes about 12 hours to finish training 30 epochs. |
Beta Was this translation helpful? Give feedback.
-
Also, 4xV100 (32 GB) was 3 days for cc12m Moving this to discussion for future reference |
Beta Was this translation helpful? Give feedback.
-
FYI, for 8 v100 GPUs, (4 nodes), it takes 2 hours per epoch for training size of 20m using RN50. @piotr-teterwak, you did not mention your training size. I think number of GPUs is more important than number of workers. If your GPU is 100% already, further increasing number of workers won't help. To speed up, you will want to increase the number of GPUs. |
Beta Was this translation helpful? Give feedback.
-
How long is a reasonable time for an epoch using 8 workers? I'm seeing about 8 hours/epoch, for the resnet50. Launch command from the README:
Thank you!
Beta Was this translation helpful? Give feedback.
All reactions