Spikely loss after each epoc

[haideraltahan]

I'm trying to reproduce the ViT-B32 results with 512 V100 GPUs and batch size ~65k on LIAON400M. However, I'm facing this issue where my loss would spike after each epoch and my zero-shot imagenet accuracy would be <1%. @rom1504 have you had this issue before?

[rwightman]

AFAIK we've never seen loss spikes synched with epochs like that, oscillations ala #822 that seem to line up with epochs aren't uncommon ... but dont' appear to have a significant impact and are nowhere near as significant as this based all the runs various people associated with this project have done.