Meet RAdam - imo the new state of the art AI optimizer

[Seb]

@a_yasyrev

And, by the way - i try script from examples on 1 GPUs and get same results as on leaderboard.

I wrote a post where I explain why it looks like current baseline on the leaderboard is lower than it should be. It wasn’t quite about number of GPUs after all.

ImageNette/Woof Leaderboards - guidelines for proving new high scores?

I finally re-figured out what the issue with current baseline is. I will detail it here, and I think you will see why I’ve been telling people to be cautious when comparing new ideas to the leaderboard. Baseline runs code in train_imagenette.py [1]. Check this part of the code: bs_rat = bs/256 if gpu is not None: bs_rat = num_distrib() if not gpu: print(f’lr: {lr}; eff_lr: {lrbs_rat}; size: {size}; alpha: {alpha}; mom: {mom}; eps: {eps}’) lr *= bs_rat When I run train_imagenette on 1 GP…