Meet Mish: New Activation function, possible successor to ReLU?

[muellerzr (Zachary Mueller)]

@LessW2020

Please go ahead

Re: optimizers - Ranger was my idea, but @rwightman and @fgfm have both coded up improved implementations. I’m hoping to have time to go through theirs and update mine and we can leverage the sync control that @fgfm mentioned earlier. The reason for integrating is of course to get ready for AutoOpt integration in the future.

Got it! I’ll send in the PR and leave the optimizers to you guys

Good to hear that you’re making progress with AutoOpt! (I wonder if we should try the autoLR that’s floating on the forums… AutoLRFinder - #41 by aychang ) some more FFT!