Meet Mish: New Activation function, possible successor to ReLU?

[LessW2020 (Less)]

The EfficientNets are a great example, even at roughly 1/10 the parameter count they actually utilize as much or more GPU memory for a given performance range (accuracy).

wow, no wonder that thing was so slow when I tested it (and gave up on it). Thanks again for the info!

[muellerzr (Zachary Mueller)]

@LessW2020 @Seb I am writing a walkthrough notebook for my meetup in two weeks over this, and I’m trying to go through and gather all of the papers that were used. Where did the Flatten and Anneal originate from?

Otherwise here is what I’ve gathered so far (I’ll update this post here in case anyone wants a quick reference to the papers):

Papers Referenced:

• Bag of Tricks for Resnet (aka the birth of xResNet)
• Large Batch Optimization for Deep Learning, LAMB
• Large Batch Training of Convolutional Networks, LARS
• Lookahead Optimizer: k steps forward, 1 step back
• Mish: A Self Regularized Non-Monotonic Neural Activation Function
• On the Variance of the Adaptive Learning Rate and Beyond, RAdam
• Self-Attention Generative Adversarial Networks
• Stochastic Gradient Methods with Layer-wise
  Adaptive Moments for Training of Deep Networks, Novograd

Other Equally as Important Noteables:

• Flatten + Anneal Scheduling - Mikhail Grankin
• Simple Self Attention - Seb

[Seb]

@grankin came up with flat+anneal I believe.
simpleselfattention is inspired by Self-Attention GANs, I highly modified their layer, and came up with the positioning in xresnet that we used. Maybe I’ll write a paper if I find a good use to it. https://github.com/sdoria/SimpleSelfAttention (I need to improve that readme)
( I should add that @grankin implemented a “symmetrical” version which we didn’t use here and participated in the testing)

A big jump from the leaderboard was fixing that learning rate oversight in learn.py
Another one was from using the full size dataset rather than Imagewoof160
Another smaller one was adding more channels in xresnet.
I think you got the rest.

[LessW2020 (Less)]

Where did the Flatten and Anneal originate from?

@grankin invented that

[muellerzr (Zachary Mueller)]

Thanks Seb! I appreciate the double check. @LessW2020 thanks for the post! I believe you missed the LARS paper though

I will certainly reference him twice then. Once I have the notebook written up, I can post it on your forum if you think it would be nice Less. I won’t do the 5 for 5 like we have been, it’s colab so it’ll just be 1 run of 5 for each effort

I also just put an “Other Equally as Important Noteables” for the non-papers

[LessW2020 (Less)]

I made the promised post to try and provide an overview for everyone on the new techniques we’ve been using here:

How we beat the 5 epoch ImageWoof leaderboard score - some new techniques to consider

Hi all, I’m very happy to announce that after a lot of work, we’ve finally been able to pretty soundly improve the leaderboard score for 5 epoch Imagewoof. It took a lot of new techniques combined and instead of reading the long Mish activation function thread where most of this happend, I thought it might be better to try and summarize here. This way, you can quickly see an overview of new techniques that we had success with, and see if they might be of use in your own work and you’ll be awa…

re: missed paper - Good spot @muellerzr - I’ve added the LARS link!

Re: notebook - yes, please add it to the github repo, that will be a nice add for sure. Thanks for making that list of papers, that’s a big help for anyone to delve into more details.

@Seb - I had to stretch to summarize the self attention aspect in my post, so I’ve referenced you in that thread for people to ask for a tutorial about it It does look promising though after seeing the results here and a quick read of the paper.

[LessW2020 (Less)]

By the way, do we have the annealing fit using the callback system done? (Or are we still calling it via 6 lines) I was thinking of trying to recreate it as a callback for an eventual merge with the library (if it seems fit) like fit one cycle

(Or worst case practice)

Still calling it with 6 lines - a one line function call for it would be awesome!

[muellerzr (Zachary Mueller)]

Thanks Less! I will work on it eventually this week, as converting scheduler to callback is…a welcome challenge. I’m following this for the scheduler, but I think if I follow the fit_one_cycle code I should be fine

[LessW2020 (Less)]

simpleselfattention is inspired by Self-Attention GANs, I highly modified their layer, and came up with the positioning in xresnet that we used. Maybe I’ll write a paper if I find a good use to it.

I would vote for you to write a paper on it. Especially if we can continue to show it’s advantages testing on more datasets, etc.

[muellerzr (Zachary Mueller)]

I would vote for you to write a paper on it. Especially if we can continue to show it’s advantages testing on more datasets, etc.

Seconded!

[muellerzr (Zachary Mueller)]

Just FFT, did we ever attempt MixUp with this? If not, why not? (as in did we just not get to it or is there reasoning)

[Seb]

If I see results that show we can train Imagenette/woof to convergence faster with ssa on different image sizes, then a paper would make sense. So far I’ve only seen that on Imagewoof128 and it didn’t work (equal results) on Imagewoof256. Weird!

Re: Mixup. I’ve just gone by Jeremy’s intuition on the leaderboard. He uses Mixup for 80 epochs and more. When I did runs on 80 epochs, I used it.

[LessW2020 (Less)]

Just FFT, did we ever attempt MixUp with this? If not, why not? (as in did we just not get to it or is there reasoning)

I did briefly test with it but similar to @Seb, I figured if Jeremy wasn’t using it then it wasn’t a high priority.

That said, I did see consistently better short term validation results with it (i.e. val curve was much more below the training curve) vs not, but at the same time at least with OneCycle, I didn’t end up any more accurate.

So I think it’s worth testing now that we have the new lr schedule and Ranger…and for that matter, I think progressive sprinkles is another thing to test as I had really good luck with that (better than cutmix usually).

[muellerzr (Zachary Mueller)]

Thanks for the input guys!

Last little bit then I think I’m good to go. Did I miss any of our runs that were especially important here:

• Baseline (Adam + xResnet50) + OneCycle
• Ranger (RAdam + LookAhead) + OneCycle
• Ranger + Flatten Anneal
• Ranger + MXResnet (xResnet50 + Mish) + Flatten Anneal
• RangerLars (Ralamb + LARS + Ranger) + MXResnet + Flatten Anneal
• RangerLars + xResnet50 + Flatten Anneal
• Ranger + SimpleSelfAttention + MXResnet + Flatten Anneal

There was quite a few so trying to get the most important <10

*Edit: forgot to add Ranger to the SSA

[muellerzr (Zachary Mueller)]

@LessW2020 I put in the PR. Let me know when it’s good and I’ll post on the other forum!

[LessW2020 (Less)]

It’s in, thanks!

[LessW2020 (Less)]

I made a new simple® setup for flat+cosine, fcfit:

#flat and cosine annealer - @mgrankin invented
#let's make it fast and easy - @lessw2020

def fcfit(learn, num_epoch=2, lr=4e-3, start_pct=.72, f_show_curve=True):
    if num_epoch<1:
        raiseValueError("num_epoch must be 1 or higher")
    n = len(learn.data.train_dl)
    anneal_start = int(n*num_epoch*start_pct) #compute what batch to start
    batch_finish = (n*num_epoch - anneal_start)
    phase0 = TrainingPhase(anneal_start).schedule_hp('lr', lr)
    phase1 = TrainingPhase(n*5 - anneal_start).schedule_hp('lr', lr, anneal=annealing_cos)
    phases = [phase0, phase1]
    sched = GeneralScheduler(learn, phases)
    #save the setup
    learn.callbacks.append(sched)
    #start the training
    print(f"fcfit: num_epochs: {num_epoch}, lr = {lr}") 
    print(f"Flat for {anneal_start} epochs, then cosine anneal for {batch_finish}")
    learn.fit(num_epoch)
    #bonus -show lr curve?
    if f_show_curve:    
        learn.recorder.plot_lr()   

looks like this:

I added it as fcfit.py to the repo.

Let’s improve it and then see about getting it into FastAI so we can just call learn.fcfit()

[muellerzr (Zachary Mueller)]

Great job @LessW2020! I’ve been busy busting out that notebook so haven’t gotten there yet… I still have a few hours at work, let me see what I can do

[LessW2020 (Less)]

Awesome, hope you can tweak and improve it. Clearly, the whole flat + cosine has had a big impact with the new optimizers.

I’m also wondering about how the accuracy variance jumps for example on epochs 8-15 (on average) with the 20 epochs and wondering if we need another bump up or down in the middle to help that.
Then I also think that time is probably better spent working on getting AutoOpt up and running so we don’t need to hand tune lol.

[muellerzr (Zachary Mueller)]

I was thinking the same. FFT for others perhaps. I’ll mention it at my meetup and perhaps get some other students who want to explore this.

If AutoOpt lives up to its name, perhaps it would be better